This document compares RabbitMQ and Apache Kafka messaging systems. It provides an overview of core concepts for each including queues/topics, exchanges/partitions, and consumer groups. It also includes example messaging patterns and topologies for handling orders in an e-commerce system, demonstrating how each system could be used to implement request/response and publish-subscribe messaging across services.
2. Background
• Jack Vanlightly
• Software Architect of SII Concatel, Barcelona
• Event-Driven Architectures
• Messaging Systems
• Data Integrations
• https://jack-vanlightly.com
4. The Log
(Apache Kafka)
The Queue
(RabbitMQ)
The Queue
• First In First Out data structure
• Optimized for efficient write and read operations from either end of
the sequence
• Stored data is transient in nature
• Not shared between applications
7 6 5 4 3 2 1Write Read
VS The Log
• Also known as Write-Ahead Logs, Transaction Logs and Commit
Logs.
• Append-only. Ordered By Time.
• Unique log sequence numbers.
• Stored data is persistent in nature
• Shared between applications
1 2 3 4 5 6 7
Reader
Reader
5. RabbitMQ
The Building Blocks
Routing
Fanout Exchange
Direct Exchange
Topic Exchange
Header Exchange
Consistent Hashing Exchange
Random Exchange
Sharding
Queues and Consumers
Queues
Competing Consumers
Non-Competing Consumers
Ephemeral Queues
Lazy Queues
Priority Queues
More
Deadletter Exchange
Alternate Exchange
Virtual Hosts
6. Exchanges, Queues and Bindings
• Publishers send messages to Exchanges
• Exchanges route messages to Queues and even other
Exchanges.
• Consumers read from Queues
• Bindings link Exchanges to Queues and even Exchanges to
Exchanges
Publisher Exchange Queue Consumer
binding
RabbitMQ
Building Blocks
Traditional queues on
steroids
8. Publish – Subscribe with Routing
Direct Exchange
Routing by Routing Key
Exact match routing key = binding key
Publisher
Direct
Exchange
Queue B Consumer
Queue A
Queue C Consumer
Consumer
created
modified
cancelled
Routing Key
Binding Key: created
Binding Key: modified
Binding Key: cancelled
9. Publish – Subscribe with Routing
Topic Exchange
Wildcard routing by Routing Key
* = match 1 word
# = match multiple words
booking.baggage.added
booking.baggage.removed
booking.passenger.added
booking.passenger.removed
payment.received
Publisher
Topic
Exchange
Queue B Consumer
Queue A
Queue C Consumer
Consumer
Routing Key
booking.#
*.passenger.*
payment.received
10. Publish – Subscribe with Routing
Header Exchange
Routing by message headers
Publisher
Header
Exchange
Queue B Consumer
Queue A
Queue C Consumer
Consumer
VIP=true, Country=UK, All
VIP=False
VIP: true
Country: UK
VIP=true, Country=UK, Any
11. Point-to-Point Messaging
Default Exchange
Direct exchange type
All queues have implicit binding to the default exchange.
Routes messages by Routing Key to Queue Name
Publisher
Default
Exchange
Orders Consumer
Notifications
Shipping Consumer
Consumer
Routing Key
12. Protecting Against Routing Failures
Alternate Exchange
Not an exchange type but a configured relationship
between two exchanges.
Route unrouteable messages to an alternate exchange
Publisher
Topic
Exchange
Queue Consumer
Queue
Unrouteable
messages
Consumer
Consumer
created
modified
Fanout
Exchangeunrouteable
13. Scaling Out Consumers
Fanout
Exchange
Shipping Shipping
Non-Competing Consumers
Each consumer receives every message
Scale Out with Competing Consumers
Each Shipping consumer receives a subset of the messages.
Notifications
Billing Billing
Notifier
Sales
Publisher
Fanout
Exchange
Shipping Shipping
Shipping
Shipping
Notifications Notifier
Billing Billing
Sales
Publisher
14. Scaling Out Queues
Random Exchange
Load balance messages across queues randomly
Publisher
Random
Exchange
Shipping.2 Consumer
Shipping.1
Shipping.3 Consumer
Consumer
15. Scaling Out Queues With Data Locality
Consistent Hashing Exchange
Partition messages across queues via hashing function over routing key, message header or message property
Publisher
Hashing
Exchange
Shipping.2 Consumer
Shipping.1
Shipping.3 Consumer
Consumer
Routing Key
1/3 hash space
1/3 hash space
1/3 hash space
16. Scaling Out Queues With Data Locality
Sharding Plugin and Modulus Exchange
Partition messages across queues over multiple hosts via hashing function on the routing key.
Publisher
Modulus
Exchange
Host 1 Shard 2 Consumer
Host 1 Shard 1
Host 1 Shard 3 Consumer
Consumer
Routing Key
Publisher
Modulus
Exchange
Host 2 Shard 2 Consumer
Host 2 Shard 1
Host 2 Shard 3 Consumer
Consumer
Routing Key
17. Apply limits to queues
• Length
• Size
• Time limits (TTL)
Eject messages from a queue when:
• A queue size/length limit reached.
• A message has spent longer than the TTL time limit in the
queue
Route to a deadletter exchange to avoid message loss.
Queue Limits
and
Deadletter
Exchanges
7 6 5 4 3 2 1
X
Queue
Exchange
Consumer
Queue
18. Ephemeral
• Auto-Delete Queue
• Queue Expiration (TTL)
• Exclusive Queues
• Auto-Delete Exchanges
Lazy Queues
• Memory optimized queues.
Priority Queues
• No longer FIFO.
• Publisher sets priority on messages
• Priority Queue moves higher priority messages ahead of lower
• Drawbacks – blocked low priority messages, low priority can
eject high priority when queue is full
Ephemeral Exchanges
and Queues
Lazy Queues
Priority
Queues
19. • Allows Multi-Tenant architecture
• Isolation of groups of exchanges, queues and users
• No routing between two virtual hosts
Virtual Hosts
21. • Distributed because Kafka is deployed as a cluster of nodes, for
both fault tolerance and scale
• Partitioned because each topic can be sharded into multiple
partitions, with each partition storing a disjoint set of the data.
• Replicated because partitions are usually replicated across
multiple nodes (servers).
• Commit Log because messages are stored in an append only
fashion.
Apache Kafka
Core Concepts
The Rise of the
Distributed,
Partitioned,
Replicated,
Commit Log
22. • Kafka has a simple Publish Subscribe model
• Topics
• Each Topic has one or more partitions
• Each partition has a replication factor of one or more
Topics,
Partitions
and
Consumer
Groups
Producer ConsumerTopic
Producer ConsumerPartition 2
Partition 1
Partition 3
27. • Producers append messages to the end of partitions.
• The partition is chosen by the producer (round robin or partition
key).
• Messages are removed according to the data retention policy.
• Consumers keep track their position in the partitions they
consume from – called the consumer offset
Topics,
Partitions
and
Consumer
Groups
1 2 3 4 5 6 7
Producer
Consumer 1
Write
Read at offsetData retention policyX
28. Topics,
Partitions
and
Consumer
Groups
1 2 3 4 5 6 7
Producer
Consumer 1
CG ARead at offset
Consumer 4
CG BRead at offset
Partition 1
1 2 3 4 5 6 7
Consumer 2
CG ARead at offset
Consumer 5
CG B
Read
at offset
Partition 28
1 2 3 4 5 6
Consumer 3
CG ARead at offset
Consumer 6
CG BRead at offset
Partition 3
30. Log Compaction
• Clean up old versions of a record (by record key). Keep latest
version only.
• Periodic process
• Kafka as an event store
Apache Kafka
Log Compaction
A X G U D W J X E A E O P X
A X
G U
D
W J
X E
A E O D X
G U W J A E O D X
31. Avro Schema Registry
• Send messages in a compact binary format – Avro
• Schema Registry service automates sharing of Avro schemas
between producers and consumers
• Schema Registry to safely control evolution of schemas
Apache Kafka
Schema Registry
Producer ConsumerTopic
Schema
Registry
Send
schema
Get
schema
32. Data Replication / Event Sourcing
• Kafka as a single source of truth
• Sits at the center of your architecture
• Kafka Connect
Apache Kafka
Data Replication
Event Sourcing
Kafka Topics
34. Example Scenario
eCommerce
Site
Sales and
Inventory
Billing
Shipping
PlaceOrder OrderPlaced
OrderBilledOrderPlaced
eCommerce
Site
Sales and
Inventory
Billing
Shipping
ModifyOrder OrderModified
ModificationBilledOrderModified
eCommerce
Site
Sales and
Inventory
Billing
Shipping
CancelOrder OrderCancelled
OrderCancelled
35. Example Scenario – RabbitMQ Fanout Exchanges
eCommerce
Site
PlaceOrder
SalesSales
OrderPlaced
BillingBilling
Shipping Shipping
OrderRefunded
Fanout Exchange per Event, Single Queue per Application
ModifyOrder
CancelOrder
OrderModified
OrderCancelled
OrderBilled
Modification
Billed
38. Example Scenario – Kafka Topics
eCommerce
Site
SalesOrder Requests Topic
Orders Topic
Related Events in One Topic
Billing
Shipping
OrderPlaced Event
OrderModified Event
OrderCancelled EventOrderBilled Event
ModificationBilled Event
OrderRefunded Event
OrderShipped Event
OrderShippingCancelled Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
39. Example Scenario – Kafka Topics
Related Events in a Few Topics
eCommerce
Site
SalesOrder Requests Topic
Orders Topic
Billing
Shipping
OrderPlaced Event
OrderModified Event
OrderCancelled Event
OrderBilled Event
ModificationBilled Event
OrderRefunded Event
OrderShipped Event
OrderShippingCancelled Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
Billing Topic
Shipments Topic
40. Example Scenario – Kafka Topics
One Topic per Event
eCommerce
Site
SalesModify Order Requests Topic
OrdersModified Topic
Billing
Shipping
OrderPlaced Event
OrderModified Event
OrderCancelled Event
OrderShipped Event
OrderShippingCancelled Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
Order Refunded Topic
Shipments Topic
Order Billed Topic
ModificationBilled Topic
OrdersPlaced Topic
OrdersCancelled Topic
CancelledShipments Topic
OrderRefunded Event
OrderBilled Event
ModificationBilled Event
Place Order Requests Topic
Cancel Order Requests Topic
41. Modifying the Scenario – Class of Service
eCommerce
Site
Sales and
Inventory
Billing
Shipping
PlaceOrder-Priority
PlaceOrder-Standard
OrderPlaced-Priority
OrderPlaced-Standard
OrderBilled-Priority
OrderBilled-Standard
OrderPlaced-Priority
OrderPlaced-Standard
eCommerce
Site
Sales and
Inventory
Billing
Shipping
ModifyOrder-Priority
ModifyOrder-Standard
OrderModified-Priority
OrderModified-Standard
ModificationBilled-Priority
ModificationBilled-Standard
OrderModified-Priority
OrderModified-Standard
42. Scenario #2 – RabbitMQ Topic Exchanges
eCommerce
Site
PlaceOrder
Sales
Sales-Standard
OrderPlaced
Billing-Priority
Billing
Shipping
Shipping-Priority
OrderRefunded
Direct Exchange per Event, Routing Key with Class of Service,
Queue per Class of Service
ModifyOrder
CancelOrder
OrderModified
OrderCancelled
OrderBilled
Modification
Billed
Sales-Priority
Billing-Standard
Shipping-Standard
priority
standard
43. Scenario #2 – RabbitMQ Two Layered Topic Exchanges
eCommerce
Site
amq.topic
Sales
Sales-Standard
Shipping-Priority
place.order.standard
modify.order.standard
cancel.order.standard
Billing-Priority
Billing
order.placed.priority
order.modified.priority
order.cancelled.priority
Single Topic Exchange (amq.topic)
Routing Key with event name
Class of Service as Message Header
Sales-Priority
Billing-Standard
order.placed.*
order.modified.*
order.cancelled.*
Shipping-Standard
order.placed.*
order.modified.*
order.cancelled.*
order.billed.*
modification.billed.*
Shipping
Sales (Topic)
place.order.*
modify.order.*
cancel.order.*
Billing
(Topic)
Shipping
(Topic)
order.billed.standard
modification.billed.standard
order.refunded.standard
order.billed.priority
modification.billed.priority
order.refunded.priority
#.priority
#.standard
#.priority
#.priority
#.standard
#.standard
place.order.priority
modify.order.priority
cancel.order.priority
order.placed.standard
order.modified.standard
order.cancelled.standard
44. Scenario #2 – RabbitMQ Topic and Header Exchanges
eCommerce
Site
amq.topic
Sales
Sales-Standard
Shipping-Priority
place.order
modify.order
cancel.order
Billing-Priority
Billing
order.placed
order.modified
order.cancelled
Single Topic Exchange (amq.topic)
Routing Key with event name
Class of Service as Message Header
Sales-Priority
Billing-Standard
order.placed
order.modified
order.cancelled
Shipping-Standard
order.placed
order.modified
order.cancelled,
order.billed
modification.billed
Shipping
Sales
(Header)
place.order
modify.order
cancel.order
Billing
(Header)
Shipping
(Header)order.billed
modification.billed
order.refunded
45. Scenario #2 – RabbitMQ Topic Exchanges
Priority VHost
Replicate preferred routing topology without Class of Service,
in two separate virtual hosts
Standard VHost
46. Scenario #2 – Kafka Topics
eCommerce
Site
Sales-Priority
Order Requests Topic
Orders Topic
Related Events in One Topic – Dedicated Application Instances
Billing-Priority
Shipping-Standard
OrderPlaced Event
OrderModified Event
OrderCancelled Event
OrderBilled Event
ModificationBilled Event
OrderRefunded Event
OrderShipped Event
OrderShippingCancelled Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
Sales-Standard
Billing-Standard
Shipping-Priority
47. Scenario #2 – Kafka Topics
eCommerce
Site
Priority Orders Topic
Related Events in Class of Service Based Topics – Dedicated Instances
Billing-Priority
Shipping-Standard
OrderPlaced Event
OrderModified Event
OrderCancelled Event
OrderBilled Event, ModificationBilled Event, OrderRefunded Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
Sales-Priority
Shipping-Priority
Billing-Standard
Standard Orders Topic
OrderShipped Event, OrderShippingCancelled Event
OrderShipped Event, OrderShippingCancelled Event
OrderBilled Event, ModificationBilled Event, OrderRefunded Event
Priority Order
Requests Topic
Sales-Standard
Standard Order
Requests Topic OrderPlaced Event
OrderModified Event
OrderCancelled Event
48. Scenario #2 – Replicate Events to Other Systems
eCommerce
Site
Sales-Priority
Order Requests Topic
Orders Topic
Kafka Topic as an Event Store and Data Replication Source
Billing-Priority
Shipping-Standard
Sales-Standard
Billing-Standard
Shipping-Priority
49. Decoupling of publishers from subscribers
Endpoint Decoupling
RabbitMQ: The endpoint we publish to is decoupled from
the endpoint we consume from.
Kafka: The endpoint we publish to is the same as we
subscribe from.
Temporal Decoupling
RabbitMQ: Consumers read a message once, most likely
close to the time of publishing.
Kafka: Consumers are decoupled temporally from the
publisher. Consumers can read messages multiple times,
and go back in history to read old messages.
50. RabbitMQ KafkaSimple Pub Sub
Easily Evolvable
Topologies
Transient Messages
Persistent Messages
AMQP
Time Travel
Data Replication
Event Store
Simple Scaling
Flexible Message
Routing