Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Практика Akka Streams
Алексей Романчук
2
Обо мне
3
4
Обо мне
• Знаете синтаксис Scala
• Знакомы со стандартной библиотекой Scala
• Слышали про Reactive Streams
• Видели Akka Streams*
...
В предыдущих сериях
T
T
Iterable[T]
T
Iterable[T]
Future[T]
T
Iterable[T]
Future[T]
Source[T]
• Реализуют спецификацию Reactive Streams
• Основаны на Akka
• Умеют Backpressure
• Статически типизированы
8
Akka Streams
9
Ключевые абстракции
Source SinkFlow
Базовые комбинаторы
11
Source
Source.single(1)

Source.failed(new Exception())

Source.empty[Int]



Source.repeat(1)

Source.fromIterator(() ...
11
Source
Source.single(1)

Source.failed(new Exception())

Source.empty[Int]



Source.repeat(1)

Source.fromIterator(() ...
11
Source
Source.single(1)

Source.failed(new Exception())

Source.empty[Int]



Source.repeat(1)

Source.fromIterator(() ...
11
Source
Source.single(1)

Source.failed(new Exception())

Source.empty[Int]



Source.repeat(1)

Source.fromIterator(() ...
11
Source
Source.single(1)

Source.failed(new Exception())

Source.empty[Int]



Source.repeat(1)

Source.fromIterator(() ...
11
Source
Source.single(1)

Source.failed(new Exception())

Source.empty[Int]



Source.repeat(1)

Source.fromIterator(() ...
11
Source
Source.single(1)

Source.failed(new Exception())

Source.empty[Int]



Source.repeat(1)

Source.fromIterator(() ...
11
Source
Source.single(1)

Source.failed(new Exception())

Source.empty[Int]



Source.repeat(1)

Source.fromIterator(() ...
11
Source
Source.single(1)

Source.failed(new Exception())

Source.empty[Int]



Source.repeat(1)

Source.fromIterator(() ...
12
Sink
Sink.ignore

Sink.cancelled

Sink.foreach[Int](println)



Sink.head

Sink.headOption

Sink.last

Sink.lastOption
...
12
Sink
Sink.ignore

Sink.cancelled

Sink.foreach[Int](println)



Sink.head

Sink.headOption

Sink.last

Sink.lastOption
...
12
Sink
Sink.ignore

Sink.cancelled

Sink.foreach[Int](println)



Sink.head

Sink.headOption

Sink.last

Sink.lastOption
...
12
Sink
Sink.ignore

Sink.cancelled

Sink.foreach[Int](println)



Sink.head

Sink.headOption

Sink.last

Sink.lastOption
...
12
Sink
Sink.ignore

Sink.cancelled

Sink.foreach[Int](println)



Sink.head

Sink.headOption

Sink.last

Sink.lastOption
...
12
Sink
Sink.ignore

Sink.cancelled

Sink.foreach[Int](println)



Sink.head

Sink.headOption

Sink.last

Sink.lastOption
...
12
Sink
Sink.ignore

Sink.cancelled

Sink.foreach[Int](println)



Sink.head

Sink.headOption

Sink.last

Sink.lastOption
...
13
Flow
Flow[Int].map(identity)

Flow[Int].mapAsync(10)(Future.successful)

Flow[Int].mapAsyncUnordered(10)(Future.success...
13
Flow
Flow[Int].map(identity)

Flow[Int].mapAsync(10)(Future.successful)

Flow[Int].mapAsyncUnordered(10)(Future.success...
13
Flow
Flow[Int].map(identity)

Flow[Int].mapAsync(10)(Future.successful)

Flow[Int].mapAsyncUnordered(10)(Future.success...
13
Flow
Flow[Int].map(identity)

Flow[Int].mapAsync(10)(Future.successful)

Flow[Int].mapAsyncUnordered(10)(Future.success...
13
Flow
Flow[Int].map(identity)

Flow[Int].mapAsync(10)(Future.successful)

Flow[Int].mapAsyncUnordered(10)(Future.success...
13
Flow
Flow[Int].map(identity)

Flow[Int].mapAsync(10)(Future.successful)

Flow[Int].mapAsyncUnordered(10)(Future.success...
13
Flow
Flow[Int].map(identity)

Flow[Int].mapAsync(10)(Future.successful)

Flow[Int].mapAsyncUnordered(10)(Future.success...
13
Flow
Flow[Int].map(identity)

Flow[Int].mapAsync(10)(Future.successful)

Flow[Int].mapAsyncUnordered(10)(Future.success...
14
Flow
Flow[Int].flatMapConcat(n =>

Source.fromIterator(() => Iterator.range(1, n))

)
1 2 3 Х
14
Flow
Flow[Int].flatMapConcat(n =>

Source.fromIterator(() => Iterator.range(1, n))

)
1 2 3 Х
1 Х
14
Flow
Flow[Int].flatMapConcat(n =>

Source.fromIterator(() => Iterator.range(1, n))

)
1 2 3 Х
1 Х 1 2 Х
14
Flow
Flow[Int].flatMapConcat(n =>

Source.fromIterator(() => Iterator.range(1, n))

)
1 2 3 Х
1 Х 1 2 Х 1 2 3 Х
14
Flow
Flow[Int].flatMapConcat(n =>

Source.fromIterator(() => Iterator.range(1, n))

)
1 2 3 Х
1 Х 1 2 Х 1 2 3 Х
1 1 2 1...
Пример №1
• Прочитать с хадупа логи транзакций и найти там 100
примеров фродных транзакций
16
Задача
List Files APIRead Line Result
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
17
Наивная реализация
def checkFraud(in: String): Future[Option[TransactionId]]
= ???
val result: List[TransactionId] = (f...
Ок?
• Не терять лишнего времени на скачивание
• Параллельные запросы к сервису
• Не превышать ограничения в 10 запросов в секу...
20
Красивая картинка со схемой
List Files APIRead Line Result
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
21
Красивый код на стримах
val stream =

Source

.fromIterator { () => hdfs.listStatusIterator(path) }

.filter(_.isFile)
...
Все на стандартных
комбинаторах
23
Запуск
val result: Future[List[TransactionId]] = stream.run()


val result2: Future[List[TransactionId]] = stream.run()
24
Запуск
List Files APIRead Line Result
24
Запуск
List Files APIRead Line Result
run()
Future Stream
24
Запуск
List Files APIRead Line Result
run()
Future Stream
run()
Future Stream
Материализация
Чертеж → Деталь
27
RunnableGraph
RunnableGraph[+Mat] {
def run(): Mat
}
28
Mat
final class Source[+Out, +Mat]
final class Flow[-In, +Out, +Mat]
final class Sink[-In, +Mat]
29
Sources
case object NotUsed
def single[T](element: T)
def empty[T]
def failed[T](cause: Throwable)
def repeat[T](elemen...
30
Sinks
def ignore: Sink[Any, Future[Done]]
def foreach[T](f: T => Unit): Sink[T, Future[Done]]
def head[T]: Sink[T, Futu...
30
Sinks
def ignore: Sink[Any, Future[Done]]
def foreach[T](f: T => Unit): Sink[T, Future[Done]]
def head[T]: Sink[T, Futu...
30
Sinks
def ignore: Sink[Any, Future[Done]]
def foreach[T](f: T => Unit): Sink[T, Future[Done]]
def head[T]: Sink[T, Futu...
30
Sinks
def ignore: Sink[Any, Future[Done]]
def foreach[T](f: T => Unit): Sink[T, Future[Done]]
def head[T]: Sink[T, Futu...
30
Sinks
def ignore: Sink[Any, Future[Done]]
def foreach[T](f: T => Unit): Sink[T, Future[Done]]
def head[T]: Sink[T, Futu...
30
Sinks
def ignore: Sink[Any, Future[Done]]
def foreach[T](f: T => Unit): Sink[T, Future[Done]]
def head[T]: Sink[T, Futu...
30
Sinks
def ignore: Sink[Any, Future[Done]]
def foreach[T](f: T => Unit): Sink[T, Future[Done]]
def head[T]: Sink[T, Futu...
30
Sinks
def ignore: Sink[Any, Future[Done]]
def foreach[T](f: T => Unit): Sink[T, Future[Done]]
def head[T]: Sink[T, Futu...
31
Композиция
val source: Source[Int, NotUsed] = Source.single(1)

val sink: Sink[Any, Future[Done]] = Sink.ignore



val ...
31
Композиция
val source: Source[Int, NotUsed] = Source.single(1)

val sink: Sink[Any, Future[Done]] = Sink.ignore



val ...
31
Композиция
val source: Source[Int, NotUsed] = Source.single(1)

val sink: Sink[Any, Future[Done]] = Sink.ignore



val ...
31
Композиция
val source: Source[Int, NotUsed] = Source.single(1)

val sink: Sink[Any, Future[Done]] = Sink.ignore



val ...
31
Композиция
val source: Source[Int, NotUsed] = Source.single(1)

val sink: Sink[Any, Future[Done]] = Sink.ignore



val ...
31
Композиция
val source: Source[Int, NotUsed] = Source.single(1)

val sink: Sink[Any, Future[Done]] = Sink.ignore



val ...
32
Композиция
33
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =
Source

.fromIterator { () => hdfs.lis...
33
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =
Source

.fromIterator { () => hdfs.lis...
33
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =
Source

.fromIterator { () => hdfs.lis...
33
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =
Source

.fromIterator { () => hdfs.lis...
34
Красивая картинка со схемой
List Files APIRead Line Result
run()
Stream
run()
Future StreamFuture
34
Красивая картинка со схемой
List Files APIRead Line Result
run()
Future
Stream
run()
Future StreamFuture
Пример №2
• Читать данные из очереди и обрабатывать.
36
Задача
Source Process Sink
Future
37
Красивый код на стримах
val source = ZMQSource(context,

mode = ZMQ.PULL,

timeout = 1 second,

addresses = List("tcp: ...
37
Красивый код на стримах
val source = ZMQSource(context,

mode = ZMQ.PULL,

timeout = 1 second,

addresses = List("tcp: ...
37
Красивый код на стримах
val source = ZMQSource(context,

mode = ZMQ.PULL,

timeout = 1 second,

addresses = List("tcp: ...
37
Красивый код на стримах
val source = ZMQSource(context,

mode = ZMQ.PULL,

timeout = 1 second,

addresses = List("tcp: ...
38
Красивый код на стримах
val stream = source

.mapAsync(10)(process)

.toMat(Sink.last)(Keep.right)



val result: Futur...
38
Красивый код на стримах
val stream = source

.mapAsync(10)(process)

.toMat(Sink.last)(Keep.right)



val result: Futur...
38
Красивый код на стримах
val stream = source

.mapAsync(10)(process)

.toMat(Sink.last)(Keep.right)



val result: Futur...
38
Красивый код на стримах
val stream = source

.mapAsync(10)(process)

.toMat(Sink.last)(Keep.right)



val result: Futur...
38
Красивый код на стримах
val stream = source

.mapAsync(10)(process)

.toMat(Sink.last)(Keep.right)



val result: Futur...
39
Красивая картинка со схемой
Source Process Sink
Future
39
Красивая картинка со схемой
Source Process Sink
Future
StreamFuture
Как это остановить?
41
Проблема остановки
val stream = source

.mapAsync(10)(process)

.toMat(Sink.last)(Keep.right)



val result: Future[Int...
42
Проблема остановки
Source Process Sink
Future
42
Проблема остановки
Source Process Sink
FutureControl
42
Проблема остановки
Source Process Sink
Future
StreamControl
Control
Future
43
Проблема остановки
trait Control {

/**

* Disconnect the underlying ZMQ socket, deliver the
remaining data and finally...
44
Проблема остановки
val source: Source[ByteString, Control] = ???
val stream: RunnableGraph[(Control, Future[Int])] =
so...
44
Проблема остановки
val source: Source[ByteString, Control] = ???
val stream: RunnableGraph[(Control, Future[Int])] =
so...
44
Проблема остановки
val source: Source[ByteString, Control] = ???
val stream: RunnableGraph[(Control, Future[Int])] =
so...
44
Проблема остановки
val source: Source[ByteString, Control] = ???
val stream: RunnableGraph[(Control, Future[Int])] =
so...
44
Проблема остановки
val source: Source[ByteString, Control] = ???
val stream: RunnableGraph[(Control, Future[Int])] =
so...
44
Проблема остановки
val source: Source[ByteString, Control] = ???
val stream: RunnableGraph[(Control, Future[Int])] =
so...
Пример №3
46
Механика
Flow[T]
46
Механика
Flow[T]
Request
Cancel
46
Механика
Flow[T]
Request Request
Cancel Cancel
46
Механика
Flow[T]
Request Request
T
Error
Complete
Cancel Cancel
46
Механика
Flow[T]
T
Error
Complete
Request Request
T
Error
Complete
Cancel Cancel
• Emit элемента
• Завершение
• Ошибка
• Backpressure
• Отмена
47
Что нужно знать про stages
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
48
Свой Source
val s: Source[Int, ActorRef] =

Source.actorPublisher[Int](Props(new MySourceActor))



class MySourceActor...
49
Reactive Zmq
case object GracefulStop

case object DeliverMore

class MySourceActor(addr: String)
extends ActorPublishe...
49
Reactive Zmq
case object GracefulStop

case object DeliverMore

class MySourceActor(addr: String)
extends ActorPublishe...
49
Reactive Zmq
case object GracefulStop

case object DeliverMore

class MySourceActor(addr: String)
extends ActorPublishe...
49
Reactive Zmq
case object GracefulStop

case object DeliverMore

class MySourceActor(addr: String)
extends ActorPublishe...
49
Reactive Zmq
case object GracefulStop

case object DeliverMore

class MySourceActor(addr: String)
extends ActorPublishe...
49
Reactive Zmq
case object GracefulStop

case object DeliverMore

class MySourceActor(addr: String)
extends ActorPublishe...
49
Reactive Zmq
case object GracefulStop

case object DeliverMore

class MySourceActor(addr: String)
extends ActorPublishe...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
50
Reactive Zmq
override def receive: Receive = {

case ActorPublisherMessage.Request(_) | DeliverMore =>

Option(conn.rec...
51
Reactive Zmq
val source: Source[Array[Byte], Control] =

Source.actorPublisher[Array[Byte]](

Props(new MySourceActor( ...
51
Reactive Zmq
val source: Source[Array[Byte], Control] =

Source.actorPublisher[Array[Byte]](

Props(new MySourceActor( ...
51
Reactive Zmq
val source: Source[Array[Byte], Control] =

Source.actorPublisher[Array[Byte]](

Props(new MySourceActor( ...
51
Reactive Zmq
val source: Source[Array[Byte], Control] =

Source.actorPublisher[Array[Byte]](

Props(new MySourceActor( ...
51
Reactive Zmq
val source: Source[Array[Byte], Control] =

Source.actorPublisher[Array[Byte]](

Props(new MySourceActor( ...
51
Reactive Zmq
val source: Source[Array[Byte], Control] =

Source.actorPublisher[Array[Byte]](

Props(new MySourceActor( ...
51
Reactive Zmq
val source: Source[Array[Byte], Control] =

Source.actorPublisher[Array[Byte]](

Props(new MySourceActor( ...
52
Свой Sink
val s = Sink.actorSubscriber[Int](Props(new MySinkActor))



class MySinkActor extends ActorSubscriber {

ove...
52
Свой Sink
val s = Sink.actorSubscriber[Int](Props(new MySinkActor))



class MySinkActor extends ActorSubscriber {

ove...
52
Свой Sink
val s = Sink.actorSubscriber[Int](Props(new MySinkActor))



class MySinkActor extends ActorSubscriber {

ove...
52
Свой Sink
val s = Sink.actorSubscriber[Int](Props(new MySinkActor))



class MySinkActor extends ActorSubscriber {

ove...
52
Свой Sink
val s = Sink.actorSubscriber[Int](Props(new MySinkActor))



class MySinkActor extends ActorSubscriber {

ove...
52
Свой Sink
val s = Sink.actorSubscriber[Int](Props(new MySinkActor))



class MySinkActor extends ActorSubscriber {

ove...
52
Свой Sink
val s = Sink.actorSubscriber[Int](Props(new MySinkActor))



class MySinkActor extends ActorSubscriber {

ove...
52
Свой Sink
val s = Sink.actorSubscriber[Int](Props(new MySinkActor))



class MySinkActor extends ActorSubscriber {

ove...
52
Свой Sink
val s = Sink.actorSubscriber[Int](Props(new MySinkActor))



class MySinkActor extends ActorSubscriber {

ove...
Итоги
• Стандартные комбинаторы
• Материализация
• Управление жизненным циклом
• Свои stages
54
Итоги
Серебряная пуля
• Более простая технология не подходит
• Backpressure
• Упорядоченные потоки
• Локальность
• Фиксированная топология
56
Ко...
• Документация Akka Streams
• Альпака
• reactive-zmq
• reactive-kafka
• Первое видео про Akka Streams
• Слайды этой презен...
Вопросы?
alexey.romanchuk@gmail.com @1esha
Backpressure
60
Событий слишком много
60
Событий слишком много
Отправитель
60
Событий слишком много
Отправитель Получатель
60
Событий слишком много
Отправитель Получатель
61
Событий слишком много
Отправитель Получатель
62
Событий слишком много
Отправитель Получатель
63
Событий слишком много
Отправитель Получатель
64
Событий слишком много
Отправитель Получатель
65
Событий слишком много
Отправитель Получатель
66
Событий слишком много
Отправитель Получатель
67
Событий слишком много
Отправитель Получатель
68
Блокирующий вызов
Отправитель Получатель
68
Блокирующий вызов
Отправитель Получатель
68
Блокирующий вызов
Отправитель Получатель
69
Pull
Отправитель Получатель
69
Pull
Отправитель Получатель
69
Pull
Отправитель Получатель
70
Negative Acknowledge
Отправитель Получатель
70
Negative Acknowledge
Отправитель Получатель
70
Negative Acknowledge
Отправитель Получатель
71
Negative Acknowledge
Отправитель Получатель
72
Dynamic pull-push
Отправитель Получатель
72
Dynamic pull-push
Отправитель Получатель
4
73
Dynamic pull-push
4
Отправитель Получатель
74
Dynamic pull-push
3
Отправитель Получатель
75
Dynamic pull-push
3
Отправитель Получатель
76
Dynamic pull-push
0
Отправитель Получатель
77
Dynamic pull-push
0
Отправитель Получатель
78
Dynamic pull-push
0
Отправитель Получатель
78
Dynamic pull-push
0
Отправитель Получатель
2
79
Dynamic pull-push
2
Отправитель Получатель
80
Dynamic pull-push
0
Отправитель Получатель
81
Balance
Balance
81
Balance
Balance
81
Balance
Balance
82
Broadcast
Broadcast
82
Broadcast
Broadcast
82
Broadcast
Broadcast
83
Zip
Zip
83
Zip
Zip
83
Zip
Zip
84
Unzip
Unzip
84
Unzip
Unzip
84
Unzip
Unzip
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
85
Красивый код на стримах
val stream: RunnableGraph[Future[List[TransactionId]]] =

Source

.fromIterator { () => hdfs.li...
Prochain SlideShare
Chargement dans…5
×

0

Partager

Télécharger pour lire hors ligne

«Практическое применение Akka Streams» — Алексей Романчук, 2ГИС

Télécharger pour lire hors ligne

Выступление на FPCONF-2016

  • Soyez le premier à aimer ceci

«Практическое применение Akka Streams» — Алексей Романчук, 2ГИС

  1. 1. Практика Akka Streams Алексей Романчук
  2. 2. 2 Обо мне
  3. 3. 3
  4. 4. 4 Обо мне
  5. 5. • Знаете синтаксис Scala • Знакомы со стандартной библиотекой Scala • Слышали про Reactive Streams • Видели Akka Streams* 5 О вас
  6. 6. В предыдущих сериях
  7. 7. T
  8. 8. T Iterable[T]
  9. 9. T Iterable[T] Future[T]
  10. 10. T Iterable[T] Future[T] Source[T]
  11. 11. • Реализуют спецификацию Reactive Streams • Основаны на Akka • Умеют Backpressure • Статически типизированы 8 Akka Streams
  12. 12. 9 Ключевые абстракции Source SinkFlow
  13. 13. Базовые комбинаторы
  14. 14. 11 Source Source.single(1)
 Source.failed(new Exception())
 Source.empty[Int]
 
 Source.repeat(1)
 Source.fromIterator(() => Iterator(1, 2, 3))
 Source.cycle(() => Iterator(1, 2, 3))
 
 Source.tick(1 second, 100 millis, ())
  15. 15. 11 Source Source.single(1)
 Source.failed(new Exception())
 Source.empty[Int]
 
 Source.repeat(1)
 Source.fromIterator(() => Iterator(1, 2, 3))
 Source.cycle(() => Iterator(1, 2, 3))
 
 Source.tick(1 second, 100 millis, ())
  16. 16. 11 Source Source.single(1)
 Source.failed(new Exception())
 Source.empty[Int]
 
 Source.repeat(1)
 Source.fromIterator(() => Iterator(1, 2, 3))
 Source.cycle(() => Iterator(1, 2, 3))
 
 Source.tick(1 second, 100 millis, ())
  17. 17. 11 Source Source.single(1)
 Source.failed(new Exception())
 Source.empty[Int]
 
 Source.repeat(1)
 Source.fromIterator(() => Iterator(1, 2, 3))
 Source.cycle(() => Iterator(1, 2, 3))
 
 Source.tick(1 second, 100 millis, ())
  18. 18. 11 Source Source.single(1)
 Source.failed(new Exception())
 Source.empty[Int]
 
 Source.repeat(1)
 Source.fromIterator(() => Iterator(1, 2, 3))
 Source.cycle(() => Iterator(1, 2, 3))
 
 Source.tick(1 second, 100 millis, ())
  19. 19. 11 Source Source.single(1)
 Source.failed(new Exception())
 Source.empty[Int]
 
 Source.repeat(1)
 Source.fromIterator(() => Iterator(1, 2, 3))
 Source.cycle(() => Iterator(1, 2, 3))
 
 Source.tick(1 second, 100 millis, ())
  20. 20. 11 Source Source.single(1)
 Source.failed(new Exception())
 Source.empty[Int]
 
 Source.repeat(1)
 Source.fromIterator(() => Iterator(1, 2, 3))
 Source.cycle(() => Iterator(1, 2, 3))
 
 Source.tick(1 second, 100 millis, ())
  21. 21. 11 Source Source.single(1)
 Source.failed(new Exception())
 Source.empty[Int]
 
 Source.repeat(1)
 Source.fromIterator(() => Iterator(1, 2, 3))
 Source.cycle(() => Iterator(1, 2, 3))
 
 Source.tick(1 second, 100 millis, ())
  22. 22. 11 Source Source.single(1)
 Source.failed(new Exception())
 Source.empty[Int]
 
 Source.repeat(1)
 Source.fromIterator(() => Iterator(1, 2, 3))
 Source.cycle(() => Iterator(1, 2, 3))
 
 Source.tick(1 second, 100 millis, ())
  23. 23. 12 Sink Sink.ignore
 Sink.cancelled
 Sink.foreach[Int](println)
 
 Sink.head
 Sink.headOption
 Sink.last
 Sink.lastOption
 
 Sink.fold(0)(_ + _)
 Sink.reduce[Int](_ + _)
  24. 24. 12 Sink Sink.ignore
 Sink.cancelled
 Sink.foreach[Int](println)
 
 Sink.head
 Sink.headOption
 Sink.last
 Sink.lastOption
 
 Sink.fold(0)(_ + _)
 Sink.reduce[Int](_ + _)
  25. 25. 12 Sink Sink.ignore
 Sink.cancelled
 Sink.foreach[Int](println)
 
 Sink.head
 Sink.headOption
 Sink.last
 Sink.lastOption
 
 Sink.fold(0)(_ + _)
 Sink.reduce[Int](_ + _)
  26. 26. 12 Sink Sink.ignore
 Sink.cancelled
 Sink.foreach[Int](println)
 
 Sink.head
 Sink.headOption
 Sink.last
 Sink.lastOption
 
 Sink.fold(0)(_ + _)
 Sink.reduce[Int](_ + _)
  27. 27. 12 Sink Sink.ignore
 Sink.cancelled
 Sink.foreach[Int](println)
 
 Sink.head
 Sink.headOption
 Sink.last
 Sink.lastOption
 
 Sink.fold(0)(_ + _)
 Sink.reduce[Int](_ + _)
  28. 28. 12 Sink Sink.ignore
 Sink.cancelled
 Sink.foreach[Int](println)
 
 Sink.head
 Sink.headOption
 Sink.last
 Sink.lastOption
 
 Sink.fold(0)(_ + _)
 Sink.reduce[Int](_ + _)
  29. 29. 12 Sink Sink.ignore
 Sink.cancelled
 Sink.foreach[Int](println)
 
 Sink.head
 Sink.headOption
 Sink.last
 Sink.lastOption
 
 Sink.fold(0)(_ + _)
 Sink.reduce[Int](_ + _)
  30. 30. 13 Flow Flow[Int].map(identity)
 Flow[Int].mapAsync(10)(Future.successful)
 Flow[Int].mapAsyncUnordered(10)(Future.successful)
 
 Flow[Int].filter(_ % 2 == 0)
 Flow[Int].collect {
 case n if n % 2 == 0 => n
 } Flow[Int].take(10)
 Flow[Int].takeWhile(_ > 10)
 Flow[Int].drop(10)
 Flow[Int].dropWhile(_ > 10)
  31. 31. 13 Flow Flow[Int].map(identity)
 Flow[Int].mapAsync(10)(Future.successful)
 Flow[Int].mapAsyncUnordered(10)(Future.successful)
 
 Flow[Int].filter(_ % 2 == 0)
 Flow[Int].collect {
 case n if n % 2 == 0 => n
 } Flow[Int].take(10)
 Flow[Int].takeWhile(_ > 10)
 Flow[Int].drop(10)
 Flow[Int].dropWhile(_ > 10)
  32. 32. 13 Flow Flow[Int].map(identity)
 Flow[Int].mapAsync(10)(Future.successful)
 Flow[Int].mapAsyncUnordered(10)(Future.successful)
 
 Flow[Int].filter(_ % 2 == 0)
 Flow[Int].collect {
 case n if n % 2 == 0 => n
 } Flow[Int].take(10)
 Flow[Int].takeWhile(_ > 10)
 Flow[Int].drop(10)
 Flow[Int].dropWhile(_ > 10)
  33. 33. 13 Flow Flow[Int].map(identity)
 Flow[Int].mapAsync(10)(Future.successful)
 Flow[Int].mapAsyncUnordered(10)(Future.successful)
 
 Flow[Int].filter(_ % 2 == 0)
 Flow[Int].collect {
 case n if n % 2 == 0 => n
 } Flow[Int].take(10)
 Flow[Int].takeWhile(_ > 10)
 Flow[Int].drop(10)
 Flow[Int].dropWhile(_ > 10)
  34. 34. 13 Flow Flow[Int].map(identity)
 Flow[Int].mapAsync(10)(Future.successful)
 Flow[Int].mapAsyncUnordered(10)(Future.successful)
 
 Flow[Int].filter(_ % 2 == 0)
 Flow[Int].collect {
 case n if n % 2 == 0 => n
 } Flow[Int].take(10)
 Flow[Int].takeWhile(_ > 10)
 Flow[Int].drop(10)
 Flow[Int].dropWhile(_ > 10)
  35. 35. 13 Flow Flow[Int].map(identity)
 Flow[Int].mapAsync(10)(Future.successful)
 Flow[Int].mapAsyncUnordered(10)(Future.successful)
 
 Flow[Int].filter(_ % 2 == 0)
 Flow[Int].collect {
 case n if n % 2 == 0 => n
 } Flow[Int].take(10)
 Flow[Int].takeWhile(_ > 10)
 Flow[Int].drop(10)
 Flow[Int].dropWhile(_ > 10)
  36. 36. 13 Flow Flow[Int].map(identity)
 Flow[Int].mapAsync(10)(Future.successful)
 Flow[Int].mapAsyncUnordered(10)(Future.successful)
 
 Flow[Int].filter(_ % 2 == 0)
 Flow[Int].collect {
 case n if n % 2 == 0 => n
 } Flow[Int].take(10)
 Flow[Int].takeWhile(_ > 10)
 Flow[Int].drop(10)
 Flow[Int].dropWhile(_ > 10)
  37. 37. 13 Flow Flow[Int].map(identity)
 Flow[Int].mapAsync(10)(Future.successful)
 Flow[Int].mapAsyncUnordered(10)(Future.successful)
 
 Flow[Int].filter(_ % 2 == 0)
 Flow[Int].collect {
 case n if n % 2 == 0 => n
 } Flow[Int].take(10)
 Flow[Int].takeWhile(_ > 10)
 Flow[Int].drop(10)
 Flow[Int].dropWhile(_ > 10)
  38. 38. 14 Flow Flow[Int].flatMapConcat(n =>
 Source.fromIterator(() => Iterator.range(1, n))
 ) 1 2 3 Х
  39. 39. 14 Flow Flow[Int].flatMapConcat(n =>
 Source.fromIterator(() => Iterator.range(1, n))
 ) 1 2 3 Х 1 Х
  40. 40. 14 Flow Flow[Int].flatMapConcat(n =>
 Source.fromIterator(() => Iterator.range(1, n))
 ) 1 2 3 Х 1 Х 1 2 Х
  41. 41. 14 Flow Flow[Int].flatMapConcat(n =>
 Source.fromIterator(() => Iterator.range(1, n))
 ) 1 2 3 Х 1 Х 1 2 Х 1 2 3 Х
  42. 42. 14 Flow Flow[Int].flatMapConcat(n =>
 Source.fromIterator(() => Iterator.range(1, n))
 ) 1 2 3 Х 1 Х 1 2 Х 1 2 3 Х 1 1 2 1 2 3 Х
  43. 43. Пример №1
  44. 44. • Прочитать с хадупа логи транзакций и найти там 100 примеров фродных транзакций 16 Задача List Files APIRead Line Result
  45. 45. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  46. 46. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  47. 47. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  48. 48. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  49. 49. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  50. 50. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  51. 51. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  52. 52. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  53. 53. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  54. 54. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  55. 55. 17 Наивная реализация def checkFraud(in: String): Future[Option[TransactionId]] = ??? val result: List[TransactionId] = (for {
 file <- hdfs.listStatusIterator(path)
 if file.isFile
 line <-fromInputStream(fs.open(file.getPath)).getLines()
 resultOpt = Await.result(checkFraud(line), 10 seconds)
 result <- resultOpt
 } yield result) .take(100).toList
  56. 56. Ок?
  57. 57. • Не терять лишнего времени на скачивание • Параллельные запросы к сервису • Не превышать ограничения в 10 запросов в секунду 19 Требования
  58. 58. 20 Красивая картинка со схемой List Files APIRead Line Result
  59. 59. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  60. 60. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  61. 61. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  62. 62. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  63. 63. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  64. 64. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  65. 65. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  66. 66. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  67. 67. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  68. 68. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  69. 69. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  70. 70. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  71. 71. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  72. 72. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  73. 73. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  74. 74. 21 Красивый код на стримах val stream =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  75. 75. Все на стандартных комбинаторах
  76. 76. 23 Запуск val result: Future[List[TransactionId]] = stream.run() 
 val result2: Future[List[TransactionId]] = stream.run()
  77. 77. 24 Запуск List Files APIRead Line Result
  78. 78. 24 Запуск List Files APIRead Line Result run() Future Stream
  79. 79. 24 Запуск List Files APIRead Line Result run() Future Stream run() Future Stream
  80. 80. Материализация
  81. 81. Чертеж → Деталь
  82. 82. 27 RunnableGraph RunnableGraph[+Mat] { def run(): Mat }
  83. 83. 28 Mat final class Source[+Out, +Mat] final class Flow[-In, +Out, +Mat] final class Sink[-In, +Mat]
  84. 84. 29 Sources case object NotUsed def single[T](element: T) def empty[T] def failed[T](cause: Throwable) def repeat[T](element: T) def fromIterator[T](f: () => Iterator[T]) def cycle[T](f: () => Iterator[T])
  85. 85. 30 Sinks def ignore: Sink[Any, Future[Done]] def foreach[T](f: T => Unit): Sink[T, Future[Done]] def head[T]: Sink[T, Future[T]] def headOption[T]: Sink[T, Future[Option[T]]] def fold[U, T](z: U)(f: (U, T) => U): Sink[T, Future[U]] def reduce[T](f: (T, T) => T): Sink[T, Future[T]]
  86. 86. 30 Sinks def ignore: Sink[Any, Future[Done]] def foreach[T](f: T => Unit): Sink[T, Future[Done]] def head[T]: Sink[T, Future[T]] def headOption[T]: Sink[T, Future[Option[T]]] def fold[U, T](z: U)(f: (U, T) => U): Sink[T, Future[U]] def reduce[T](f: (T, T) => T): Sink[T, Future[T]]
  87. 87. 30 Sinks def ignore: Sink[Any, Future[Done]] def foreach[T](f: T => Unit): Sink[T, Future[Done]] def head[T]: Sink[T, Future[T]] def headOption[T]: Sink[T, Future[Option[T]]] def fold[U, T](z: U)(f: (U, T) => U): Sink[T, Future[U]] def reduce[T](f: (T, T) => T): Sink[T, Future[T]]
  88. 88. 30 Sinks def ignore: Sink[Any, Future[Done]] def foreach[T](f: T => Unit): Sink[T, Future[Done]] def head[T]: Sink[T, Future[T]] def headOption[T]: Sink[T, Future[Option[T]]] def fold[U, T](z: U)(f: (U, T) => U): Sink[T, Future[U]] def reduce[T](f: (T, T) => T): Sink[T, Future[T]]
  89. 89. 30 Sinks def ignore: Sink[Any, Future[Done]] def foreach[T](f: T => Unit): Sink[T, Future[Done]] def head[T]: Sink[T, Future[T]] def headOption[T]: Sink[T, Future[Option[T]]] def fold[U, T](z: U)(f: (U, T) => U): Sink[T, Future[U]] def reduce[T](f: (T, T) => T): Sink[T, Future[T]]
  90. 90. 30 Sinks def ignore: Sink[Any, Future[Done]] def foreach[T](f: T => Unit): Sink[T, Future[Done]] def head[T]: Sink[T, Future[T]] def headOption[T]: Sink[T, Future[Option[T]]] def fold[U, T](z: U)(f: (U, T) => U): Sink[T, Future[U]] def reduce[T](f: (T, T) => T): Sink[T, Future[T]]
  91. 91. 30 Sinks def ignore: Sink[Any, Future[Done]] def foreach[T](f: T => Unit): Sink[T, Future[Done]] def head[T]: Sink[T, Future[T]] def headOption[T]: Sink[T, Future[Option[T]]] def fold[U, T](z: U)(f: (U, T) => U): Sink[T, Future[U]] def reduce[T](f: (T, T) => T): Sink[T, Future[T]]
  92. 92. 30 Sinks def ignore: Sink[Any, Future[Done]] def foreach[T](f: T => Unit): Sink[T, Future[Done]] def head[T]: Sink[T, Future[T]] def headOption[T]: Sink[T, Future[Option[T]]] def fold[U, T](z: U)(f: (U, T) => U): Sink[T, Future[U]] def reduce[T](f: (T, T) => T): Sink[T, Future[T]]
  93. 93. 31 Композиция val source: Source[Int, NotUsed] = Source.single(1)
 val sink: Sink[Any, Future[Done]] = Sink.ignore
 
 val s1: RunnableGraph[NotUsed] = source.to(sink)
 
 val s2: RunnableGraph[Future[Done]] =
 source.toMat(sink)(Keep.right)
 
 val s3: RunnableGraph[(NotUsed, Future[Done])] =
 source.toMat(sink)(Keep.both)

  94. 94. 31 Композиция val source: Source[Int, NotUsed] = Source.single(1)
 val sink: Sink[Any, Future[Done]] = Sink.ignore
 
 val s1: RunnableGraph[NotUsed] = source.to(sink)
 
 val s2: RunnableGraph[Future[Done]] =
 source.toMat(sink)(Keep.right)
 
 val s3: RunnableGraph[(NotUsed, Future[Done])] =
 source.toMat(sink)(Keep.both)

  95. 95. 31 Композиция val source: Source[Int, NotUsed] = Source.single(1)
 val sink: Sink[Any, Future[Done]] = Sink.ignore
 
 val s1: RunnableGraph[NotUsed] = source.to(sink)
 
 val s2: RunnableGraph[Future[Done]] =
 source.toMat(sink)(Keep.right)
 
 val s3: RunnableGraph[(NotUsed, Future[Done])] =
 source.toMat(sink)(Keep.both)

  96. 96. 31 Композиция val source: Source[Int, NotUsed] = Source.single(1)
 val sink: Sink[Any, Future[Done]] = Sink.ignore
 
 val s1: RunnableGraph[NotUsed] = source.to(sink)
 
 val s2: RunnableGraph[Future[Done]] =
 source.toMat(sink)(Keep.right)
 
 val s3: RunnableGraph[(NotUsed, Future[Done])] =
 source.toMat(sink)(Keep.both)

  97. 97. 31 Композиция val source: Source[Int, NotUsed] = Source.single(1)
 val sink: Sink[Any, Future[Done]] = Sink.ignore
 
 val s1: RunnableGraph[NotUsed] = source.to(sink)
 
 val s2: RunnableGraph[Future[Done]] =
 source.toMat(sink)(Keep.right)
 
 val s3: RunnableGraph[(NotUsed, Future[Done])] =
 source.toMat(sink)(Keep.both)

  98. 98. 31 Композиция val source: Source[Int, NotUsed] = Source.single(1)
 val sink: Sink[Any, Future[Done]] = Sink.ignore
 
 val s1: RunnableGraph[NotUsed] = source.to(sink)
 
 val s2: RunnableGraph[Future[Done]] =
 source.toMat(sink)(Keep.right)
 
 val s3: RunnableGraph[(NotUsed, Future[Done])] =
 source.toMat(sink)(Keep.both)

  99. 99. 32 Композиция
  100. 100. 33 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] = Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  101. 101. 33 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] = Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  102. 102. 33 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] = Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  103. 103. 33 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] = Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  104. 104. 34 Красивая картинка со схемой List Files APIRead Line Result run() Stream run() Future StreamFuture
  105. 105. 34 Красивая картинка со схемой List Files APIRead Line Result run() Future Stream run() Future StreamFuture
  106. 106. Пример №2
  107. 107. • Читать данные из очереди и обрабатывать. 36 Задача Source Process Sink Future
  108. 108. 37 Красивый код на стримах val source = ZMQSource(context,
 mode = ZMQ.PULL,
 timeout = 1 second,
 addresses = List("tcp: //127.0.0.1:12345")
 )
 
 def process(in: ByteString): Future[Int] = ???
  109. 109. 37 Красивый код на стримах val source = ZMQSource(context,
 mode = ZMQ.PULL,
 timeout = 1 second,
 addresses = List("tcp: //127.0.0.1:12345")
 )
 
 def process(in: ByteString): Future[Int] = ???
  110. 110. 37 Красивый код на стримах val source = ZMQSource(context,
 mode = ZMQ.PULL,
 timeout = 1 second,
 addresses = List("tcp: //127.0.0.1:12345")
 )
 
 def process(in: ByteString): Future[Int] = ???
  111. 111. 37 Красивый код на стримах val source = ZMQSource(context,
 mode = ZMQ.PULL,
 timeout = 1 second,
 addresses = List("tcp: //127.0.0.1:12345")
 )
 
 def process(in: ByteString): Future[Int] = ???
  112. 112. 38 Красивый код на стримах val stream = source
 .mapAsync(10)(process)
 .toMat(Sink.last)(Keep.right)
 
 val result: Future[Int] = stream.run()
 result.onComplete(_ => "We are done here")

  113. 113. 38 Красивый код на стримах val stream = source
 .mapAsync(10)(process)
 .toMat(Sink.last)(Keep.right)
 
 val result: Future[Int] = stream.run()
 result.onComplete(_ => "We are done here")

  114. 114. 38 Красивый код на стримах val stream = source
 .mapAsync(10)(process)
 .toMat(Sink.last)(Keep.right)
 
 val result: Future[Int] = stream.run()
 result.onComplete(_ => "We are done here")

  115. 115. 38 Красивый код на стримах val stream = source
 .mapAsync(10)(process)
 .toMat(Sink.last)(Keep.right)
 
 val result: Future[Int] = stream.run()
 result.onComplete(_ => "We are done here")

  116. 116. 38 Красивый код на стримах val stream = source
 .mapAsync(10)(process)
 .toMat(Sink.last)(Keep.right)
 
 val result: Future[Int] = stream.run()
 result.onComplete(_ => "We are done here")

  117. 117. 39 Красивая картинка со схемой Source Process Sink Future
  118. 118. 39 Красивая картинка со схемой Source Process Sink Future StreamFuture
  119. 119. Как это остановить?
  120. 120. 41 Проблема остановки val stream = source
 .mapAsync(10)(process)
 .toMat(Sink.last)(Keep.right)
 
 val result: Future[Int] = stream.run()
 result.onComplete(_ => "We are done here")

  121. 121. 42 Проблема остановки Source Process Sink Future
  122. 122. 42 Проблема остановки Source Process Sink FutureControl
  123. 123. 42 Проблема остановки Source Process Sink Future StreamControl Control Future
  124. 124. 43 Проблема остановки trait Control {
 /**
 * Disconnect the underlying ZMQ socket, deliver the remaining data and finally close the socket.
 */
 def gracefulStop(): Unit
 }
  125. 125. 44 Проблема остановки val source: Source[ByteString, Control] = ??? val stream: RunnableGraph[(Control, Future[Int])] = source
 .mapAsync(10)(process)
 .toMat(Sink.head)(Keep.both)
 
 val (control: Control, result: Future[Int]) = stream.run()
 result.onComplete(_ => "We are done here")
 control.gracefulStop()
  126. 126. 44 Проблема остановки val source: Source[ByteString, Control] = ??? val stream: RunnableGraph[(Control, Future[Int])] = source
 .mapAsync(10)(process)
 .toMat(Sink.head)(Keep.both)
 
 val (control: Control, result: Future[Int]) = stream.run()
 result.onComplete(_ => "We are done here")
 control.gracefulStop()
  127. 127. 44 Проблема остановки val source: Source[ByteString, Control] = ??? val stream: RunnableGraph[(Control, Future[Int])] = source
 .mapAsync(10)(process)
 .toMat(Sink.head)(Keep.both)
 
 val (control: Control, result: Future[Int]) = stream.run()
 result.onComplete(_ => "We are done here")
 control.gracefulStop()
  128. 128. 44 Проблема остановки val source: Source[ByteString, Control] = ??? val stream: RunnableGraph[(Control, Future[Int])] = source
 .mapAsync(10)(process)
 .toMat(Sink.head)(Keep.both)
 
 val (control: Control, result: Future[Int]) = stream.run()
 result.onComplete(_ => "We are done here")
 control.gracefulStop()
  129. 129. 44 Проблема остановки val source: Source[ByteString, Control] = ??? val stream: RunnableGraph[(Control, Future[Int])] = source
 .mapAsync(10)(process)
 .toMat(Sink.head)(Keep.both)
 
 val (control: Control, result: Future[Int]) = stream.run()
 result.onComplete(_ => "We are done here")
 control.gracefulStop()
  130. 130. 44 Проблема остановки val source: Source[ByteString, Control] = ??? val stream: RunnableGraph[(Control, Future[Int])] = source
 .mapAsync(10)(process)
 .toMat(Sink.head)(Keep.both)
 
 val (control: Control, result: Future[Int]) = stream.run()
 result.onComplete(_ => "We are done here")
 control.gracefulStop()
  131. 131. Пример №3
  132. 132. 46 Механика Flow[T]
  133. 133. 46 Механика Flow[T] Request Cancel
  134. 134. 46 Механика Flow[T] Request Request Cancel Cancel
  135. 135. 46 Механика Flow[T] Request Request T Error Complete Cancel Cancel
  136. 136. 46 Механика Flow[T] T Error Complete Request Request T Error Complete Cancel Cancel
  137. 137. • Emit элемента • Завершение • Ошибка • Backpressure • Отмена 47 Что нужно знать про stages
  138. 138. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  139. 139. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  140. 140. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  141. 141. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  142. 142. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  143. 143. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  144. 144. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  145. 145. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  146. 146. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  147. 147. 48 Свой Source val s: Source[Int, ActorRef] =
 Source.actorPublisher[Int](Props(new MySourceActor))
 
 class MySourceActor extends ActorPublisher[Int] {
 override def receive: Receive = {
 case ActorPublisherMessage.Request(n) =>
 onNext(100)
 onCompleteThenStop()
 onErrorThenStop(new Exception)
 case ActorPublisherMessage.Cancel =>
 context.stop(self)
 }
 }

  148. 148. 49 Reactive Zmq case object GracefulStop
 case object DeliverMore
 class MySourceActor(addr: String) extends ActorPublisher[Array[Byte]] {
 var conn: ZMQ.Socket = ???
 
 override def preStart(): Unit = {
 conn.connect(addr)
 }
 
 override def postStop(): Unit = {
 conn.disconnect(addr)
 }

  149. 149. 49 Reactive Zmq case object GracefulStop
 case object DeliverMore
 class MySourceActor(addr: String) extends ActorPublisher[Array[Byte]] {
 var conn: ZMQ.Socket = ???
 
 override def preStart(): Unit = {
 conn.connect(addr)
 }
 
 override def postStop(): Unit = {
 conn.disconnect(addr)
 }

  150. 150. 49 Reactive Zmq case object GracefulStop
 case object DeliverMore
 class MySourceActor(addr: String) extends ActorPublisher[Array[Byte]] {
 var conn: ZMQ.Socket = ???
 
 override def preStart(): Unit = {
 conn.connect(addr)
 }
 
 override def postStop(): Unit = {
 conn.disconnect(addr)
 }

  151. 151. 49 Reactive Zmq case object GracefulStop
 case object DeliverMore
 class MySourceActor(addr: String) extends ActorPublisher[Array[Byte]] {
 var conn: ZMQ.Socket = ???
 
 override def preStart(): Unit = {
 conn.connect(addr)
 }
 
 override def postStop(): Unit = {
 conn.disconnect(addr)
 }

  152. 152. 49 Reactive Zmq case object GracefulStop
 case object DeliverMore
 class MySourceActor(addr: String) extends ActorPublisher[Array[Byte]] {
 var conn: ZMQ.Socket = ???
 
 override def preStart(): Unit = {
 conn.connect(addr)
 }
 
 override def postStop(): Unit = {
 conn.disconnect(addr)
 }

  153. 153. 49 Reactive Zmq case object GracefulStop
 case object DeliverMore
 class MySourceActor(addr: String) extends ActorPublisher[Array[Byte]] {
 var conn: ZMQ.Socket = ???
 
 override def preStart(): Unit = {
 conn.connect(addr)
 }
 
 override def postStop(): Unit = {
 conn.disconnect(addr)
 }

  154. 154. 49 Reactive Zmq case object GracefulStop
 case object DeliverMore
 class MySourceActor(addr: String) extends ActorPublisher[Array[Byte]] {
 var conn: ZMQ.Socket = ???
 
 override def preStart(): Unit = {
 conn.connect(addr)
 }
 
 override def postStop(): Unit = {
 conn.disconnect(addr)
 }

  155. 155. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  156. 156. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  157. 157. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  158. 158. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  159. 159. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  160. 160. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  161. 161. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  162. 162. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  163. 163. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  164. 164. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  165. 165. 50 Reactive Zmq override def receive: Receive = {
 case ActorPublisherMessage.Request(_) | DeliverMore =>
 Option(conn.recv()) match {
 case Some(msg) =>
 onNext(msg)
 if(totalDemand > 0) self ! DeliverMore
 case None => self ! DeliverMore
 }
 case ActorPublisherMessage.Cancel | GracefulStop =>
 conn.disconnect(addr)
 onCompleteThenStop()
 }

  166. 166. 51 Reactive Zmq val source: Source[Array[Byte], Control] =
 Source.actorPublisher[Array[Byte]](
 Props(new MySourceActor( ???))
 ).mapMaterializedValue { ref =>
 new Control {
 override def gracefulStop() = ref ! GracefulStop
 }
 }
  167. 167. 51 Reactive Zmq val source: Source[Array[Byte], Control] =
 Source.actorPublisher[Array[Byte]](
 Props(new MySourceActor( ???))
 ).mapMaterializedValue { ref =>
 new Control {
 override def gracefulStop() = ref ! GracefulStop
 }
 }
  168. 168. 51 Reactive Zmq val source: Source[Array[Byte], Control] =
 Source.actorPublisher[Array[Byte]](
 Props(new MySourceActor( ???))
 ).mapMaterializedValue { ref =>
 new Control {
 override def gracefulStop() = ref ! GracefulStop
 }
 }
  169. 169. 51 Reactive Zmq val source: Source[Array[Byte], Control] =
 Source.actorPublisher[Array[Byte]](
 Props(new MySourceActor( ???))
 ).mapMaterializedValue { ref =>
 new Control {
 override def gracefulStop() = ref ! GracefulStop
 }
 }
  170. 170. 51 Reactive Zmq val source: Source[Array[Byte], Control] =
 Source.actorPublisher[Array[Byte]](
 Props(new MySourceActor( ???))
 ).mapMaterializedValue { ref =>
 new Control {
 override def gracefulStop() = ref ! GracefulStop
 }
 }
  171. 171. 51 Reactive Zmq val source: Source[Array[Byte], Control] =
 Source.actorPublisher[Array[Byte]](
 Props(new MySourceActor( ???))
 ).mapMaterializedValue { ref =>
 new Control {
 override def gracefulStop() = ref ! GracefulStop
 }
 }
  172. 172. 51 Reactive Zmq val source: Source[Array[Byte], Control] =
 Source.actorPublisher[Array[Byte]](
 Props(new MySourceActor( ???))
 ).mapMaterializedValue { ref =>
 new Control {
 override def gracefulStop() = ref ! GracefulStop
 }
 }
  173. 173. 52 Свой Sink val s = Sink.actorSubscriber[Int](Props(new MySinkActor))
 
 class MySinkActor extends ActorSubscriber {
 override def receive: Receive = {
 case ActorSubscriberMessage.OnNext =>
 cancel()
 request(100)
 case ActorSubscriberMessage.OnComplete =>
 case ActorSubscriberMessage.OnError =>
 }
  174. 174. 52 Свой Sink val s = Sink.actorSubscriber[Int](Props(new MySinkActor))
 
 class MySinkActor extends ActorSubscriber {
 override def receive: Receive = {
 case ActorSubscriberMessage.OnNext =>
 cancel()
 request(100)
 case ActorSubscriberMessage.OnComplete =>
 case ActorSubscriberMessage.OnError =>
 }
  175. 175. 52 Свой Sink val s = Sink.actorSubscriber[Int](Props(new MySinkActor))
 
 class MySinkActor extends ActorSubscriber {
 override def receive: Receive = {
 case ActorSubscriberMessage.OnNext =>
 cancel()
 request(100)
 case ActorSubscriberMessage.OnComplete =>
 case ActorSubscriberMessage.OnError =>
 }
  176. 176. 52 Свой Sink val s = Sink.actorSubscriber[Int](Props(new MySinkActor))
 
 class MySinkActor extends ActorSubscriber {
 override def receive: Receive = {
 case ActorSubscriberMessage.OnNext =>
 cancel()
 request(100)
 case ActorSubscriberMessage.OnComplete =>
 case ActorSubscriberMessage.OnError =>
 }
  177. 177. 52 Свой Sink val s = Sink.actorSubscriber[Int](Props(new MySinkActor))
 
 class MySinkActor extends ActorSubscriber {
 override def receive: Receive = {
 case ActorSubscriberMessage.OnNext =>
 cancel()
 request(100)
 case ActorSubscriberMessage.OnComplete =>
 case ActorSubscriberMessage.OnError =>
 }
  178. 178. 52 Свой Sink val s = Sink.actorSubscriber[Int](Props(new MySinkActor))
 
 class MySinkActor extends ActorSubscriber {
 override def receive: Receive = {
 case ActorSubscriberMessage.OnNext =>
 cancel()
 request(100)
 case ActorSubscriberMessage.OnComplete =>
 case ActorSubscriberMessage.OnError =>
 }
  179. 179. 52 Свой Sink val s = Sink.actorSubscriber[Int](Props(new MySinkActor))
 
 class MySinkActor extends ActorSubscriber {
 override def receive: Receive = {
 case ActorSubscriberMessage.OnNext =>
 cancel()
 request(100)
 case ActorSubscriberMessage.OnComplete =>
 case ActorSubscriberMessage.OnError =>
 }
  180. 180. 52 Свой Sink val s = Sink.actorSubscriber[Int](Props(new MySinkActor))
 
 class MySinkActor extends ActorSubscriber {
 override def receive: Receive = {
 case ActorSubscriberMessage.OnNext =>
 cancel()
 request(100)
 case ActorSubscriberMessage.OnComplete =>
 case ActorSubscriberMessage.OnError =>
 }
  181. 181. 52 Свой Sink val s = Sink.actorSubscriber[Int](Props(new MySinkActor))
 
 class MySinkActor extends ActorSubscriber {
 override def receive: Receive = {
 case ActorSubscriberMessage.OnNext =>
 cancel()
 request(100)
 case ActorSubscriberMessage.OnComplete =>
 case ActorSubscriberMessage.OnError =>
 }
  182. 182. Итоги
  183. 183. • Стандартные комбинаторы • Материализация • Управление жизненным циклом • Свои stages 54 Итоги
  184. 184. Серебряная пуля
  185. 185. • Более простая технология не подходит • Backpressure • Упорядоченные потоки • Локальность • Фиксированная топология 56 Когда стоит использовать
  186. 186. • Документация Akka Streams • Альпака • reactive-zmq • reactive-kafka • Первое видео про Akka Streams • Слайды этой презентации - goo.gl/F7EYoE 57 Ссылки
  187. 187. Вопросы? alexey.romanchuk@gmail.com @1esha
  188. 188. Backpressure
  189. 189. 60 Событий слишком много
  190. 190. 60 Событий слишком много Отправитель
  191. 191. 60 Событий слишком много Отправитель Получатель
  192. 192. 60 Событий слишком много Отправитель Получатель
  193. 193. 61 Событий слишком много Отправитель Получатель
  194. 194. 62 Событий слишком много Отправитель Получатель
  195. 195. 63 Событий слишком много Отправитель Получатель
  196. 196. 64 Событий слишком много Отправитель Получатель
  197. 197. 65 Событий слишком много Отправитель Получатель
  198. 198. 66 Событий слишком много Отправитель Получатель
  199. 199. 67 Событий слишком много Отправитель Получатель
  200. 200. 68 Блокирующий вызов Отправитель Получатель
  201. 201. 68 Блокирующий вызов Отправитель Получатель
  202. 202. 68 Блокирующий вызов Отправитель Получатель
  203. 203. 69 Pull Отправитель Получатель
  204. 204. 69 Pull Отправитель Получатель
  205. 205. 69 Pull Отправитель Получатель
  206. 206. 70 Negative Acknowledge Отправитель Получатель
  207. 207. 70 Negative Acknowledge Отправитель Получатель
  208. 208. 70 Negative Acknowledge Отправитель Получатель
  209. 209. 71 Negative Acknowledge Отправитель Получатель
  210. 210. 72 Dynamic pull-push Отправитель Получатель
  211. 211. 72 Dynamic pull-push Отправитель Получатель 4
  212. 212. 73 Dynamic pull-push 4 Отправитель Получатель
  213. 213. 74 Dynamic pull-push 3 Отправитель Получатель
  214. 214. 75 Dynamic pull-push 3 Отправитель Получатель
  215. 215. 76 Dynamic pull-push 0 Отправитель Получатель
  216. 216. 77 Dynamic pull-push 0 Отправитель Получатель
  217. 217. 78 Dynamic pull-push 0 Отправитель Получатель
  218. 218. 78 Dynamic pull-push 0 Отправитель Получатель 2
  219. 219. 79 Dynamic pull-push 2 Отправитель Получатель
  220. 220. 80 Dynamic pull-push 0 Отправитель Получатель
  221. 221. 81 Balance Balance
  222. 222. 81 Balance Balance
  223. 223. 81 Balance Balance
  224. 224. 82 Broadcast Broadcast
  225. 225. 82 Broadcast Broadcast
  226. 226. 82 Broadcast Broadcast
  227. 227. 83 Zip Zip
  228. 228. 83 Zip Zip
  229. 229. 83 Zip Zip
  230. 230. 84 Unzip Unzip
  231. 231. 84 Unzip Unzip
  232. 232. 84 Unzip Unzip
  233. 233. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  234. 234. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  235. 235. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  236. 236. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  237. 237. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  238. 238. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  239. 239. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  240. 240. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  241. 241. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  242. 242. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  243. 243. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  244. 244. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  245. 245. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  246. 246. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  247. 247. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)

  248. 248. 85 Красивый код на стримах val stream: RunnableGraph[Future[List[TransactionId]]] =
 Source
 .fromIterator { () => hdfs.listStatusIterator(path) }
 .filter(_.isFile)
 .flatMapConcat { file =>
 val s = fromInputStream(hdfs.open(file.getPath))
 Source.fromIterator(() => s.getLines())
 }
 .throttle(10, 1 second, 0, ThrottleMode.Shaping)
 .mapAsyncUnordered(50)(checkFraud)
 .collect { case Some(value) => value }
 .take(100)
 .fold(List.empty[TransactionId])(_ :+ _)
 .toMat(Sink.head)(Keep.right)


Выступление на FPCONF-2016

Vues

Nombre de vues

89

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

52

Actions

Téléchargements

0

Partages

0

Commentaires

0

Mentions J'aime

0

×