Comparing 3 Message-oriented Middlewares, Kafka, NSQ, RabbitMQ. More globally, what is the best way to integrating MoM in our architectures.

1. #DevoxxFR
3 degrees of MoM
kafka, rabbitmq, nsq
Guillaume Arnaud
@guillarnaud
voyages-sncf.com
1

2. #DevoxxFR 2
P publisher
C consumer
A Message-oriented Middleware (MoM)
just a queue ?

3. #DevoxxFR 3
ActiveMQ
RabbitMQ
Nats
NSQ
ZeroMQ
Kafka
AmazonSQS
Disque
Aeron
Mosquitto
Beanstalk
Apollo
Nanomsg
…
JMS
Stomp
MQTT
AMQP
XMPP
…
Flume
Redis
Logstash
Syslog
…
Brokers Protocol/API Almost
https://github.com/mfornos/awesome-microservices#messaging
http://queues.io/

4. #DevoxxFR 4
ActiveMQ
RabbitMQ
Nats
NSQ
ZeroMQ
Kafka
AmazonSQS
Disque
Aeron
Mosquitto
Beanstalk
Apollo
Nanomsg
…
JMS
Stomp
MQTT
AMQP
XMPP
…
Flume
Redis
Logstash
Syslog
…
Brokers Protocol/API Almost
https://github.com/mfornos/awesome-microservices#messaging
http://queues.io/
ROUTING
DECENTRALIZED
DISTRIBUTED

5. #DevoxxFR
RabbitMQ.id
5
parents: Pivotal
age: v3.6.1 (march 2016)
license: mozilla public license
native languages: erlang/amqp
fluently spoken: python, java, ruby, .Net, nodejs
big brother: activemq

6. #DevoxxFR 6
P
C
decoupling
exchange
queue
routing/binding keys
C
exchange
routing key: devoxx.2016.newuser
binding key: devoxx.2016.*
routing
queue: devoxx2016_service

7. #DevoxxFR 7
P
C C
exchange: fanout
routing key: devoxx.2016.newuser
binding key: devoxx.2016.*
message 1
fanout
broadcast
no routing rules
message 1 message 1

8. #DevoxxFR 8
P
C C
exchange: direct
routing key: devoxx.2016.newuser
binding key: devoxx.2016.newuser
message 1
direct
routing key=binding key
message 1
queue: new_user_servicequeue: update_user_service
binding key: devoxx.2016.updateuser

9. #DevoxxFR 9
P
C C
exchange: topic
routing key: devoxx.2016.newuser
binding key: devoxx.2016.*
message 1
message 1 message 1
queue: devoxx_servicequeue: trigger_new_user
binding key: devoxx.2016.newuser
topic
wildcard
subscription

10. #DevoxxFR
RabbitMQ.model
10
exchange routing key
queue service
event type
binding key aggregator
business domain

11. #DevoxxFR
RabbitMQ.microservices
11
exchange
'user' / fanout
cv.*
microservices
routing between
exchanges and services
specify_user_action
audit
check_fraud
exchange
'cv' / topic
alert_recruiters
crop_photo
cv.photo
cv.employer.new
check_employer
exchange
'mailer' / direct
generate_and_send_mail

12. #DevoxxFR
RabbitMQ.ops
12
• virtual hosts / security
• plugins: management, ldap, mqtt, consistent hashing
• cluster/federation
• queue size/disk
• cpu

13. #DevoxxFR
RabbitMQ.admin
13
$ rabbitmq-plugins enable rabbitmq-management

14. #DevoxxFR
Kafka.id
14
parents: LinkedIn/confluent
age: v0.9.0.1 (feb. 2016)
license: apache
native language: java/scala
family: hdfs, spark streaming, storm, flume

15. #DevoxxFR 15
P publisher
C consumer
5
4
3
2
1
offset
lag
log file(s)
append only
zero copy
~ k*100 000 msg/s

16. #DevoxxFR 16
P
C
5
4
3
2
1
Partition
split the topic
partition key
4
3
2
1
keys 1, 3, 5…
keys 2, 4, 6…

17. #DevoxxFR 17
P
C
distribution
5
4
3
2
1
4
3
2
1
4
3
2
1
4
3
2
1
node 1 node 2

18. #DevoxxFR 18
replication
5
4
3
2
1
4
3
2
1
4
3
2
1
P
replica lag
partition leader
partition follower
partition follower

19. #DevoxxFR 19
P
C
5
4
3
2
1
4
3
2
1
C
Scaling
one consumer per partition
add partitions for scaling

20. #DevoxxFR 20
P
C
5
4
3
2
1
4
3
2
1
C
Scaling
one consumer per partition
add partitions for scaling
C

21. #DevoxxFR 21
P
C C
Scaling
one consumer per partition
add partitions for scaling
C

22. #DevoxxFR 22
P
C C
Group
consume same messages
~ JMS topic
C
group 1
group 2

23. #DevoxxFR
Kafka.event sourcing
23
domain
object
domain
object
event
view 1
view 2
event store

24. #DevoxxFR
Kafka.event sourcing
24
1 id:1234 my order (…)
2 id:abcd an other order
3 id:wxyz and again…
120 id:1234 my other order
…
logcompaction
offset key payload

25. #DevoxxFR
Kafka.ops
25
• clients are part of the distributed system
(heartbeat, tcp config, etc…)
• lag sender/consumer
• replica lags
• preferred replica election
• repartition master/slave
• don't forget Zookeeper
• size and time retention
• use kafka tools
~ equivalent to a NoSQL database
properties #
server 127
publisher 47
consumer 45

26. #DevoxxFR
Kafka.admin
26
https://github.com/yahoo/kafka-manager

27. #DevoxxFR
NSQ.id
27
parents: nsq.io
age: v0.3.7 (feb. 2016)
license: open source
native language: go
fluently spoken: go, python, javascript, http(!)
zodiacal sign: syslog++

28. #DevoxxFR 28
P
C
topic: server_start_event
nsqd
channel: server_subscribe
topic: server_start_event
nsqd
channel: server_subscribe
P
consumer: server_subscribe
C
nsqlookupd
http http
tcptcp

29. #DevoxxFR 29
server
nsqd
server
nsqd
server
nsqd
server
nsqd
push
pull
decentralized
local push
remote pull

30. #DevoxxFR 30
nsqd
nsqd
nsqd
haproxy
slave
daemon
reload
admin
tomcat
new_ip_port1
2
3
5
6
7
use case
service discovery
nsqd
tomcat

31. #DevoxxFR
NSQ.service discovery
31
nsqd
nsqd
nsqd
haproxy
slave
daemon
reload
admin
tomcat
new_ip_port1
2
3
5
6
7
nsqd
tomcat

32. #DevoxxFR
NSQ.ops
32
$ tree bin/
bin/
├── nsq_pubsub
├── nsq_stat
├── nsq_tail
├── nsq_to_file
├── nsq_to_http
├── nsq_to_nsq
├── nsqadmin
├── nsqd
├── nsqlookupd
└── to_nsq

33. #DevoxxFR
NSQ.ops
33
$ nsqd —help
-e2e-processing-latency-percentile value
…
-max-bytes-per-file int
-max-heartbeat-interval duration
-max-msg-size int
-max-msg-timeout duration
-max-output-buffer-size int
-max-output-buffer-timeout duration
-max-rdy-count int
-max-req-timeout duration
-mem-queue-size int
-msg-timeout string
-snappy
…

34. #DevoxxFR
MoM.diversity
34

35. #DevoxxFR
NSQ.admin
35
http://nsq.io/components/nsqadmin.html

36. #DevoxxFR
your MoM is a part
of your application
36

37. #DevoxxFR
your application is a part
of your distributed system
37

38. #DevoxxFR 38
http://www.meetup.com/fr-FR/Meetup-Messaging-Paris/

39. #DevoxxFR
Questions ?
39

40. #DevoxxFR
Thanks
40