My talk from ScalaDays 2016 in New York on May 11, 2016: Transitioning from a monolithic application to a set of microservices can help increase performance and scalability, but it can also drastically increase complexity. Layers of inter-service network calls for add latency and an increasing risk of failure where previously only local function calls existed. In this talk, I'll speak about how to tame this complexity using Apache Kafka and Reactive Streams to: - Extract non-critical processing from the critical path of your application to reduce request latency - Provide back-pressure to handle both slow and fast producers/consumers - Maintain high availability, high performance, and reliable messaging - Evolve message payloads while maintaining backwards and forwards compatibility.

1. Reducing Microservice Complexity
with Kafka and Reactive Streams
Jim Riecken

3. Reducing Microservice
Complexity with Kafka and
Reactive Streams
Senior Software Developer
Jim Riecken
@jimriecken - jim.riecken@hootsuite.com

4. @jimriecken

8. • Monolith to Microservices + Complexity
• Asynchronous Messaging
• Kafka
• Reactive Streams + Akka Streams
Agenda

9. • Details on how to set up a Kafka cluster
• In-depth tutorial on Akka Streams
Anti-Agenda

10. Monolith to
Microservices

11. M

12. Efficiency
Time

13. M
S1
S2

14. F
S1
S2
S3
S4
S5

15. Efficiency
Time
• Small
• Scalable
• Independent
• Easy to Create
• Clear ownership

16. Network Calls
• Latency
• Failure

17. ~99.5%
Reliability
99.9% 99.9% 99.9% 99.9%

18. Coordination
• Between services
• Between teams

20. Asynchronous
Messaging

21. Message Bus
Synchronous
Asynchronous

22. • Decoupling
• Pub/Sub
• Less coordination
• Additional consumers are easy
• Help scale organization
Why?

23. • Well-defined delivery semantics
• High-Throughput
• Highly-Available
• Durable
• Scalable
• Backpressure
Messaging Requirements

24. Kafka

25. • Distributed, partitioned, replicated
commit log service
• Pub/Sub messaging functionality
• Created by LinkedIn, now an
Apache open-source project
What is Kafka?

26. Producers
Kafka Brokers
Consumers

27. 0 | 1 | 2 | 3 | 4 | 5
0 | 1 | 2 | 3 | 4 | 5 | 6
0 | 1 | 2 | 3
P0
P1
P2
New Messages
Appended
Topic
Topics + Partitions

28. • Send messages to topics
• Responsible for choosing which
partition to send to
• Round-robin
• Consistent hashing based on a
message key
Producers

29. • Pull messages from topics
• Track their own offset in each
partition
Consumers

30. P0 P1 P2
1 2 3 4 5 6
Topic
Group 1 Group 2

31. How does Kafka
meet the
requirements?

32. • Hundreds of MB/s of reads/writes from
thousands of concurrent clients
• LinkedIn (2015)
• 800 billion messages per day (18 million/s
peak)
• 175 TB of data produced per day
• > 1000 servers in 60 clusters
Kafka is Fast

33. • Brokers
• All data is persisted to disk
• Partitions replicated to other nodes
• Consumers
• Start where they left off
• Producers
• Can retry - at-least-once messaging
Kafka is Resilient

34. • Capacity can be added at runtime
with zero downtime
• More servers => more disk space
• Topics can be larger than any single
node could hold
• Additional partitions can be added to
add more parallelism
Kafka is Scalable

35. • Large storage capacity
• Topic retention is a Consumer SLA
• Almost impossible for a fast
producer to overload a slow
consumer
• Allows real-time as well as batch
consumption
Kafka Helps with Back-Pressure

36. Message Data
Format

37. • Array[Byte]
• Serialization?
• JSON?
• Protocol Buffers
• Binary - Fast
• IDL - Code Generation
• Message evolution
Messages

38. Processing Data
with Reactive
Streams

39. • Standard for async stream
processing with non-blocking back-
pressure
• Subscriber signals demand to publisher
• Publisher sends no more than demand
• Low-level
• Mainly meant for library authors
Reactive Streams

40. Publisher[T] Subscriber[T]
onSubscribe(s: Subscription)
onNext(t: T)
onComplete()
onError(t: Throwable)
Subscription
subscribe(s: Subscriber[-T])
request(n: Long)
cancel()

41. Processing Data
with Akka Streams

42. • Library on top of Akka Actors and
Reactive Streams
• Process sequences of elements
using bounded buffer space
• Strongly Typed
Akka Streams

43. Flow
Source
Sink
Fan
Out
Fan
In
Concepts

44. Runnable Graph
Concepts

45. Composition

46. • Turning on the tap
• Create actors
• Open files/sockets/other resources
• Materialized values
• Source: Actor, Promise, Subscriber
• Sink: Actor, Future, Producer
Materialization

47. Reactive Kafka

48. • https://github.com/akka/reactive-kafka
• Akka Streams wrapper around Kafka
API
• Consumer Source
• Producer Sink
Reactive Kafka

49. • Sink - sends message to Kafka topic
• Flow - sends message to Kafka topic +
emits result downstream
• When the stream completes/fails the
connection to Kafka will be
automatically closed
Producer

50. • Source - pulls messages from
Kafka topics
• Offset Management
• Back-pressure
• Materialization
• Object that can stop the consumer
(and complete the stream)
Consumer

51. Simple Producer Example
implicit val system = ActorSystem("producer-test")
implicit val materializer = ActorMaterializer()
val producerSettings = ProducerSettings(
system, new ByteArraySerializer, new StringSerializer
).withBootstrapServers("localhost:9092")
Source(1 to 100)
.map(i => s"Message $i")
.map(m => new ProducerRecord[Array[Byte], String]("lower", m))
.to(Producer.plainSink(producerSettings)).run()

52. Simple Consumer Example
implicit val system = ActorSystem("producer-test")
implicit val materializer = ActorMaterializer()
val consumerSettings = ConsumerSettings(
system, new ByteArrayDeserializer, new StringDeserializer, Set("lower")
).withBootstrapServers("localhost:9092").withGroupId("test-group")
val control = Consumer.atMostOnceSource(consumerSettings.withClientId("client1"))
.map(record => record.value)
.to(Sink.foreach(v => println(v))).run()
control.stop()

53. val control = Consumer.committableSource(consumerSettings.withClientId("client1"))
.map { msg =>
val upper = msg.value.toUpperCase
Producer.Message(
new ProducerRecord[Array[Byte], String]("upper", upper),
msg.committableOffset)
}.to(Producer.commitableSink(producerSettings)).run()
control.stop()
Combined Example

54. Demo

55. Wrap-Up

56. • Microservices have many advantages, but can
introduce failure and complexity.
• Asynchronous messaging can help reduce this
complexity and Kafka is a great option.
• Akka Streams makes reliably processing data
from Kafka with back-pressure easy
Wrap-Up

58. Thank you!
Questions?
@jimriecken - jim.riecken@hootsuite.com
Jim Riecken