The project of redesigning the core event bus of the Allegro auction platform showed us a lot of issues with distributed systems on all the levels from network to data serialization. How to assure our clients that not even a single event is lost? How to respond in less than 65ms even if Kafka is rebuilding its cluster? ...And many other.

1. Krzysztof Dębski
Atmosphere 2015
PubSub++
or how to make life with Kafka easier

2. @DebskiChris Kraków, 19.05.2015, Atmosphere
Why do we need a PubSub?

3. @DebskiChris Kraków, 19.05.2015, Atmosphere
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Distributed environment

4. @DebskiChris Kraków, 19.05.2015, Atmosphere
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Service
Domain
Domain
Domain
Distributed environment

5. @DebskiChris Kraków, 19.05.2015, Atmosphere
Orchestration vs. Choreography

6. @DebskiChris Kraków, 19.05.2015, Atmosphere
Kafka

7. @DebskiChris Kraków, 19.05.2015, Atmosphere
Kafka architecture
Service
Producer
Service
Consumer
Kafka
Broker
Zookeeper
< Kafka 0.9

8. @DebskiChris Kraków, 19.05.2015, Atmosphere
Kafka data
10
9
7
8
6
5
3
4
2
1
8
5
4
2
1
10
9
7
6
3
5
4
2
8
1
Data Topic Partition

9. @DebskiChris Kraków, 19.05.2015, Atmosphere
Consumer groups
Consumer 1
Broker
P0
Broker
P1
Broker
P2
Broker
P3
Consumer 2 Consumer 3 Consumer 4 Consumer 5 Consumer 6
Consumer group 1 Consumer group 2

10. @DebskiChris Kraków, 19.05.2015, Atmosphere
Publish & subscribe
Producer_1
…
Producer_n
Consumer_group_1Consumer_group_2
Remove old messages
Remove old events
Publish event
Read eventRead event

11. @DebskiChris Kraków, 19.05.2015, Atmosphere
Leader & Replicas
Service
Producer
Service
Consumer
Broker
Zookeeper
Broker
Broker
P1 P0
P2 P1
P0 P2

12. @DebskiChris Kraków, 19.05.2015, Atmosphere
Rebalancing leaders
Broker 1
Broker 2
Broker 3
P1 P0
P2 P1
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs:
retention.ms=86400000
Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1
Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2
Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3

13. @DebskiChris Kraków, 19.05.2015, Atmosphere
Rebalancing leaders
Broker 1
Broker 2
Broker 3
P1 P0
P2 P1
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs:
retention.ms=86400000
Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1
Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2
Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3

14. @DebskiChris Kraków, 19.05.2015, Atmosphere
Rebalancing leaders
Broker 1
Broker 2
Broker 3
P1 P0
P2 P1
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs:
retention.ms=86400000
Topic: test Partition: 0 Leader: 1 Replicas: 3, 1 ISR: 1
Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2
Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2

15. @DebskiChris Kraków, 19.05.2015, Atmosphere
Rebalancing leaders
Broker 1
Broker 2
Broker 3
P1 P0
P2 P1
P0 P2
Topic: test Partition count: 3 Replication factor: 1 Configs:
retention.ms=86400000
Topic: test Partition: 0 Leader: 1 Replicas: 3, 1 ISR: 1, 3
Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2
Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3

16. @DebskiChris Kraków, 19.05.2015, Atmosphere
Acknowledges
0 - don’t wait for response from broker
1 - only leader needs to respond
-1 - all replicas in sync
Speed
Safety

17. @DebskiChris Kraków, 19.05.2015, Atmosphere
Events security
Basically, there is no authentication
You can create any topic
You can publish everywhere
You can subscribe to any topic

18. @DebskiChris Kraków, 19.05.2015, Atmosphere

19. @DebskiChris Kraków, 19.05.2015, Atmosphere
Synchronous world
Service
A
Service
B

20. @DebskiChris Kraków, 19.05.2015, Atmosphere
Asynchronous made simple
Service
A
Service
B

21. @DebskiChris Kraków, 19.05.2015, Atmosphere
Hermes
Hermes
Frontend
Hermes
Frontend
Hermes
Frontend
Hermes
Consumer
Hermes
Consumer

22. @DebskiChris Kraków, 19.05.2015, Atmosphere
Event identification
Hermes
Frontend
Kafka
Broker
POST
{“event”: ”test”}
{
"id": "58d7ff07-dd0e-4103-9b1f-55706f3049e6",
"timestamp”: 1430443071995,
“data”: {“event”: ”test”}
}
HTTP 201 Created
Message-id: 58d7ff07-dd0e-4103-9b1f-55706f3049e6

23. @DebskiChris Kraków, 19.05.2015, Atmosphere
100 ms timeout
(on the client side)

24. @DebskiChris Kraków, 19.05.2015, Atmosphere
Network
Nagle’s algorithm
42 ms -> 6 ms
High load on client servers
wrong timeout calculation

25. @DebskiChris Kraków, 19.05.2015, Atmosphere
65 ms timeout
(on the server side)

26. @DebskiChris Kraków, 19.05.2015, Atmosphere
Normal operation
Hermes
Frontend
Producer
Hermes
Consumer
Consumer
Kafka
Broker
Zookeeper
POST
HTTP
201
Created

27. @DebskiChris Kraków, 19.05.2015, Atmosphere
Slow kafka response
Hermes
Frontend
Producer
Hermes
Consumer
Consumer
Kafka
Broker
Zookeeper
POST
HTTP
202
Accepted

28. @DebskiChris Kraków, 19.05.2015, Atmosphere
Improved offset management
Hermes
Producer
Hermes consumer
Remove old messages
Publish event
Commited
Local unsent queue
Read event

29. @DebskiChris Kraków, 19.05.2015, Atmosphere
Resend messages
Hermes consumer
Remove old messages
Local unsent queue
New event
Service
instance

30. @DebskiChris Kraków, 19.05.2015, Atmosphere
Resend messages
Hermes consumer
Remove old messages
Local unsent queue
New event
Service
instance
HTTP 503
Unavailable

31. @DebskiChris Kraków, 19.05.2015, Atmosphere
Resend messages
Remove old messages
Local unsent queue
New event
Service
instance
HTTP 503
Unavailable
Check TTL &
Add to queue
Hermes consumer

32. @DebskiChris Kraków, 19.05.2015, Atmosphere
Resend speed
100% adapt 1/s 1/min

33. @DebskiChris Kraków, 19.05.2015, Atmosphere
Lost events
Hermes
Frontend
Producer
Hermes
Consumer
Consumer
Kafka
Broker
Zookeeper
Tracker

34. @DebskiChris Kraków, 19.05.2015, Atmosphere
Groups and Topics
pl.allegro.hermes.demo.basic
Group Topic

35. @DebskiChris Kraków, 19.05.2015, Atmosphere
Security
Authentication and authorization interfaces
You can create any topic in your group
You can publish everywhere
Group owner defines subscriptions

36. @DebskiChris Kraków, 19.05.2015, Atmosphere
Multi datacenter
Hermes
Frontend
Hermes
Manager
Hermes
Frontend
Hermes
Consumer
Hermes
Consumer
DC2DC1

37. @DebskiChris Kraków, 19.05.2015, Atmosphere
What’s in it for me?

38. @DebskiChris Kraków, 19.05.2015, Atmosphere
What we plan to do?

39. @DebskiChris Kraków, 19.05.2015, Atmosphere
Lower overhead
Avro
RFC 7540

40. @DebskiChris Kraków, 19.05.2015, Atmosphere
Avro
RFC 7540 => HTTP/2
Lower overhead

41. @DebskiChris Kraków, 19.05.2015, Atmosphere
Improve security
Service authentication

42. @DebskiChris Kraków, 19.05.2015, Atmosphere
What’ve we learned?

43. @DebskiChris Kraków, 19.05.2015, Atmosphere
Tradeoffs
You cannot have exactly one delivery
http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

44. @DebskiChris Kraków, 19.05.2015, Atmosphere
Clients
Not all clients are the same.

45. @DebskiChris Kraków, 19.05.2015, Atmosphere
Early adopters are your friends
You know exactly what features are needed.

46. @debskichris
allegrotech.io