My talk from ScalaDays 2016 in New York on May 11, 2016:
Transitioning from a monolithic application to a set of microservices can help increase performance and scalability, but it can also drastically increase complexity. Layers of inter-service network calls for add latency and an increasing risk of failure where previously only local function calls existed. In this talk, I'll speak about how to tame this complexity using Apache Kafka and Reactive Streams to:
- Extract non-critical processing from the critical path of your application to reduce request latency
- Provide back-pressure to handle both slow and fast producers/consumers
- Maintain high availability, high performance, and reliable messaging
- Evolve message payloads while maintaining backwards and forwards compatibility.
25. • Distributed, partitioned, replicated
commit log service
• Pub/Sub messaging functionality
• Created by LinkedIn, now an
Apache open-source project
What is Kafka?
28. • Send messages to topics
• Responsible for choosing which
partition to send to
• Round-robin
• Consistent hashing based on a
message key
Producers
29. • Pull messages from topics
• Track their own offset in each
partition
Consumers
32. • Hundreds of MB/s of reads/writes from
thousands of concurrent clients
• LinkedIn (2015)
• 800 billion messages per day (18 million/s
peak)
• 175 TB of data produced per day
• > 1000 servers in 60 clusters
Kafka is Fast
33. • Brokers
• All data is persisted to disk
• Partitions replicated to other nodes
• Consumers
• Start where they left off
• Producers
• Can retry - at-least-once messaging
Kafka is Resilient
34. • Capacity can be added at runtime
with zero downtime
• More servers => more disk space
• Topics can be larger than any single
node could hold
• Additional partitions can be added to
add more parallelism
Kafka is Scalable
35. • Large storage capacity
• Topic retention is a Consumer SLA
• Almost impossible for a fast
producer to overload a slow
consumer
• Allows real-time as well as batch
consumption
Kafka Helps with Back-Pressure
39. • Standard for async stream
processing with non-blocking back-
pressure
• Subscriber signals demand to publisher
• Publisher sends no more than demand
• Low-level
• Mainly meant for library authors
Reactive Streams
49. • Sink - sends message to Kafka topic
• Flow - sends message to Kafka topic +
emits result downstream
• When the stream completes/fails the
connection to Kafka will be
automatically closed
Producer
50. • Source - pulls messages from
Kafka topics
• Offset Management
• Back-pressure
• Materialization
• Object that can stop the consumer
(and complete the stream)
Consumer
51. Simple Producer Example
implicit val system = ActorSystem("producer-test")
implicit val materializer = ActorMaterializer()
val producerSettings = ProducerSettings(
system, new ByteArraySerializer, new StringSerializer
).withBootstrapServers("localhost:9092")
Source(1 to 100)
.map(i => s"Message $i")
.map(m => new ProducerRecord[Array[Byte], String]("lower", m))
.to(Producer.plainSink(producerSettings)).run()
52. Simple Consumer Example
implicit val system = ActorSystem("producer-test")
implicit val materializer = ActorMaterializer()
val consumerSettings = ConsumerSettings(
system, new ByteArrayDeserializer, new StringDeserializer, Set("lower")
).withBootstrapServers("localhost:9092").withGroupId("test-group")
val control = Consumer.atMostOnceSource(consumerSettings.withClientId("client1"))
.map(record => record.value)
.to(Sink.foreach(v => println(v))).run()
control.stop()
53. val control = Consumer.committableSource(consumerSettings.withClientId("client1"))
.map { msg =>
val upper = msg.value.toUpperCase
Producer.Message(
new ProducerRecord[Array[Byte], String]("upper", upper),
msg.committableOffset)
}.to(Producer.commitableSink(producerSettings)).run()
control.stop()
Combined Example
56. • Microservices have many advantages, but can
introduce failure and complexity.
• Asynchronous messaging can help reduce this
complexity and Kafka is a great option.
• Akka Streams makes reliably processing data
from Kafka with back-pressure easy
Wrap-Up
