SlideShare une entreprise Scribd logo

IBM MQ and Kafka, what is the difference?

•
•
David Ware
David Ware

Message queueing solutions used to be the one general purpose tool used for all asynchronous application patterns, then along came event streaming as an application model. To support this effectively needed a whole new approach to how messages are handled by the messaging technology. Now the tables are turned and many are wondering if an event streaming solution can be used for all their asynchronous application patterns from now on. But just as message queueing solutions work in a way to optimize for their core use cases, so do event streaming solutions, and these behaviors directly affect the applications that use them. This session picks IBM MQ and Kafka to look at how they compare and, more importantly, differ in their behavior so that you can decide which application scenarios are best suited by each. Spoiler -they're both good in their own way!

Logiciels
1  sur  56
Télécharger pour lire hors ligne
TechCon 2020 IBM MQ and Kafka, what is the difference? David Ware IBM MQ Chief Architect
Messaging is essential for building fully connected, efficient and scalable solutions. More now than ever before Messaging covers a broad spectrum of requirements
Messaging Scalability Security Management High Availability There are many capabilities essential to every enterprise messaging solution IBM MQ and Kafka have both repeatedly been proven to tick these boxes…
© 2020 IBM Corporation So just pick one and move on?
© 2020 IBM Corporation So just pick one and move on? Perhaps, if you think everything looks like a nail
© 2020 IBM Corporation So just pick one and move on? Perhaps, if you think everything looks like a nail Otherwise, let’s focus on what you’re trying to achieve…
Critical data exchange: work that needs to be done Critical applications demand assured asynchronous interactions Essential capabilities: End-to-end once and once only Deliveryü Fine grain messaging Messages typically represent commands, queries and operations The message is a way to pass control from the originator of the message to the consumer
Event Driven: building scalable microservices Microservices increases the need for communication. API-based interactions can build fragile and unscalable tight bonds between components Topics and Subscriptions Publishing and subscribing to events relaxes the coupling of microservices Events are messages that communicate that something has occurred Essential capabilities:
Event Streaming: the expanding need for messaging Event Streaming brings data together from disparate sources, enabling even more responsive and engaging experiences for a wider set of users Scalable Subscription Stream History Efficient cloud and analytics applications utilize local decoupled buffers of event data Essential capabilities:
The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
© 2020 IBM Corporation To choose the right tool you’re probably going to need to go to the next level…
© 2020 IBM Corporation Message consumption
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 … • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER 0 1 23 45 67
© 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots
© 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots Kafka topic partitions…
© 2020 IBM Corporation Queues and topics
© 2019 IBM Corporation Comparing queues and topics QUEUE 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER PRODUCER PRODUCER 0 1 23 45 6 7 TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER CONSUMER 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Queues Topics • Each messages goes to a single consumer from the queue • Each message goes to every subscriber to the topic
© 2019 IBM Corporation CONSUMERCONSUMER MQ and Kafka topics TOPIC PARTITION 0 1 2 3 4 5 6 7 CONSUMER GROUP CONSUMER GROUP PRODUCER PRODUCER CONSUMER CONSUMER TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER 1 2 3 4 5 6 70 1 2 3 4 5 6 70 IBM MQ Kafka
© 2019 IBM Corporation Queues and topics IBM MQ Kafka Queues Primary MQ concept Queues are administratively or dynamically created Queues can be partitioned for scale All messages are stored on queues, even publications n/a Topics Core MQ capability Topics are dynamically created as needed by applications Each subscribing application has access to a dedicated queue of to-be-consumed messages matching their subscription. Primary Kafka concept Rolling history of messages maintained by Kafka. Topics can be partitioned for scale… Consumer’s offset within a shared stream of messages is maintained by Kafka.
© 2020 IBM Corporation Scaling
Horizontal scaling – workload balance the messages PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER IBM MQ Kafka PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER
Kafka Clusters • A Kafka system is typically made up of multiple active nodes • Topics can be partitioned to enable workload balancing across those nodes, improving scale and availability of the topics • Kafka workload balances messages across partitions, either randomly or based on application provided context to enable per-partition ordering • Each topic partition is active in one broker at a time • The messages held in a topic partition can be replicated to other brokers to ensure data integrity across failures and automatic recovery
© 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 Kafka partitions and consumer groups 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
© 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
© 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
© 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups - p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers X
IBM MQ Clusters and workload balancing PRODUCE MESSAGES TO QUEUE A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER • A single running instance of MQ is a queue manager. • These can act independently or collectively to build horizontally scaled solutions. • An MQ Cluster is a dynamically provisioned set of queue managers with the ability to partition queues across them. • MQ will workload balance inbound messages from producing applications across each partition of the queue. • With MQ’s Uniform Cluster capability, MQ will workload balance multiple consuming applications across each queue manager, ensuring messages are consumed efficiently. • MQ does not provide simple message ordering in these topologies • MQ’s HA technologies provide data integrity across failures and automatic recovery
© 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances
© 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20, … 01,11, … X
© 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20 01,11 03,13,23
© 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20,40 01,11 03,13,23,43 CONSUMER CONSUMER CONSUMER CONSUMER 30,50,70 32,52 31 21
© 2019 IBM Corporation Scaling IBM MQ Kafka Consuming application is the bottleneck Scale out the consuming application Messaging system is the bottleneck Scale out the messaging system Increase the number of consuming applications from the existing queue/subscription. Increase the number of topic partitions to support a greater number of consuming applications in a consumer group. Increase the number of queue managers deployed and workload balance messages and consumers across them. Increase the number of Kafka broker nodes deployed and redistribute the topic partitions across them. †Good practice is to capacity plan for highest load and pre-define multiple topic partitions to support more consumers than you currently require, enabling more dynamic consumer scaling https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ † † † Unlike MQ, where new queue managers do not require copies of the existing messages, Kafka requires event histories to be copied to a new broker before it can be effective, slowing down dynamic scaling https://www.infoq.com/presentations/cloud-native-kafka-netflix/ ††
© 2020 IBM Corporation Conversational messaging
© 2019 IBM Corporation ConversationConversation ConversationConversation Fine grain communication? • MQ is designed for tens of thousands of lightweight queues and hundreds of thousands of dynamic topic strings per queue manager • This enables each conversation (e.g. reply flow) to be a dedicated queue or topic • Each consumer only sees its intended message • Dynamic create/deletion of queues and topic strings IBM MQ Equivalent Kafka model … • Kafka is designed for less, larger, configured topic partitions per broker • Recommendations are typically for 2-4k topic partitions per broker* • Consumers process every message in the topic from a point forward • This model is considered a Kafka anti-pattern. Recommendation is to rearchitect application to a stream processing pattern if possible** • https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions ** https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka PRODUCERPRODUCER PRODUCERPRODUCER CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … TOPIC 2 30 1 … …
© 2020 IBM Corporation Message delivery
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Any problem in the messaging system or the application may result in messages being lost Messages are not lost in the event of a failure, but any message may be duplicated Messages are not lost in the event of a failure and each message can be processed individually to prevent possible duplication An application can coordinate the processing of a message with a specific update to an external resource, such as a database
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications Ideal for event streaming use case Ideal for critical data exchange use case
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Higher levels of message reliability typically reduce the need for applications to consider error scenarios. However, higher message reliability typically reduces overall message performance and can limit overall availability The trade off Event streaming criticaldata exchange Reliability Performance
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Kafka IBM MQ https://medium.com/@andrew_schofield/does-apache-kafka-do-acid-transactions-647b207f3d0e Sweet spots Default behavior. Some messages may be lost following an application or Kafka failure, but others persisted Default behavior. All messages of this type lost following an MQ failure by design Kafka well designed for this, producers can be configured to retry failed sends and consumers typically periodically update their offset Possible to design an application to behave like this through message batching and protocol optimizations, but not a key MQ scenario Possible within a closed Kafka system, but much harder with external systems. Technically possible to lock Kafka down to confirm every message for individual acknowledgment but loses benefits A natural fit for the destructive consume approach of MQ coupled with its core persistence model Supports Kafka-only transactions for exactly-once stream processing applications, but no external resource coordination A core MQ capability is as a resource manager (or coordinator) for global transactions Event streaming criticaldata exchange
© 2020 IBM Corporation Message availability
© 2019 IBM Corporation Per-message availability IBM MQ Kafka Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes. PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9
© 2019 IBM Corporation Per-message availability IBM MQ Kafka PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 active/active partitioning of queues and topics, each partition is active/passive Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes.
© 2020 IBM Corporation Summary
© 2020 IBM Corporation If you want to build the best solution, you’re probably going to need more than a hammer
The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
Thank you David Ware IBM MQ Chief Architect — www.linkedin.com/in/dware1 © Copyright IBM Corporation 2020. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and ibm.com are trademarks of IBM Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available at Copyright and trademark information. 57TechCon 2020 © 2020 IBM Corporation
TechCon 2020 / © 2020 IBM Corporation 58

Recommandé

IBM MQ - What's new in 9.2
IBM MQ - What's new in 9.2IBM MQ - What's new in 9.2
IBM MQ - What's new in 9.2
David Ware
 
Building an Active-Active IBM MQ System
Building an Active-Active IBM MQ SystemBuilding an Active-Active IBM MQ System
Building an Active-Active IBM MQ System
matthew1001
 
IBM MQ - High Availability and Disaster Recovery
IBM MQ - High Availability and Disaster RecoveryIBM MQ - High Availability and Disaster Recovery
IBM MQ - High Availability and Disaster Recovery
MarkTaylorIBM
 
IBM MQ High Availability 2019
IBM MQ High Availability 2019IBM MQ High Availability 2019
IBM MQ High Availability 2019
David Ware
 
IBM Integration Bus High Availability Overview
IBM Integration Bus High Availability OverviewIBM Integration Bus High Availability Overview
IBM Integration Bus High Availability Overview
Peter Broadhurst
 
Deploying and managing IBM MQ in the Cloud
Deploying and managing IBM MQ in the CloudDeploying and managing IBM MQ in the Cloud
Deploying and managing IBM MQ in the Cloud
Robert Parker
 
IBM MQ Whats new - including 9.3 and 9.3.1
IBM MQ Whats new - including 9.3 and 9.3.1IBM MQ Whats new - including 9.3 and 9.3.1
IBM MQ Whats new - including 9.3 and 9.3.1
Robert Parker
 
IBM Think 2018: IBM MQ High Availability
IBM Think 2018: IBM MQ High AvailabilityIBM Think 2018: IBM MQ High Availability
IBM Think 2018: IBM MQ High Availability
Jamie Squibb
 

Recommandé

IBM MQ - What's new in 9.2
IBM MQ - What's new in 9.2IBM MQ - What's new in 9.2
IBM MQ - What's new in 9.2
David Ware
 
Building an Active-Active IBM MQ System
Building an Active-Active IBM MQ SystemBuilding an Active-Active IBM MQ System
Building an Active-Active IBM MQ System
matthew1001
 
IBM MQ - High Availability and Disaster Recovery
IBM MQ - High Availability and Disaster RecoveryIBM MQ - High Availability and Disaster Recovery
IBM MQ - High Availability and Disaster Recovery
MarkTaylorIBM
 
IBM MQ High Availability 2019
IBM MQ High Availability 2019IBM MQ High Availability 2019
IBM MQ High Availability 2019
David Ware
 
IBM Integration Bus High Availability Overview
IBM Integration Bus High Availability OverviewIBM Integration Bus High Availability Overview
IBM Integration Bus High Availability Overview
Peter Broadhurst
 
Deploying and managing IBM MQ in the Cloud
Deploying and managing IBM MQ in the CloudDeploying and managing IBM MQ in the Cloud
Deploying and managing IBM MQ in the Cloud
Robert Parker
 
IBM MQ Whats new - including 9.3 and 9.3.1
IBM MQ Whats new - including 9.3 and 9.3.1IBM MQ Whats new - including 9.3 and 9.3.1
IBM MQ Whats new - including 9.3 and 9.3.1
Robert Parker
 
IBM Think 2018: IBM MQ High Availability
IBM Think 2018: IBM MQ High AvailabilityIBM Think 2018: IBM MQ High Availability
IBM Think 2018: IBM MQ High Availability
Jamie Squibb
 
Fault tolerant and scalable ibm mq
Fault tolerant and scalable ibm mqFault tolerant and scalable ibm mq
Fault tolerant and scalable ibm mq
David Ware
 
Mq presentation
Mq presentationMq presentation
Mq presentation
xddu
 
IBM MQ: Using Publish/Subscribe in an MQ Network
IBM MQ: Using Publish/Subscribe in an MQ NetworkIBM MQ: Using Publish/Subscribe in an MQ Network
IBM MQ: Using Publish/Subscribe in an MQ Network
David Ware
 
Kafka with IBM Event Streams - Technical Presentation
Kafka with IBM Event Streams - Technical PresentationKafka with IBM Event Streams - Technical Presentation
Kafka with IBM Event Streams - Technical Presentation
Winton Winton
 
Whats new in MQ V9.1
Whats new in MQ V9.1Whats new in MQ V9.1
Whats new in MQ V9.1
David Ware
 
IBM MQ Overview (IBM Message Queue)
IBM MQ Overview (IBM Message Queue)IBM MQ Overview (IBM Message Queue)
IBM MQ Overview (IBM Message Queue)
Juarez Junior
 
IBM MQ High Availabillity and Disaster Recovery (2017 version)
IBM MQ High Availabillity and Disaster Recovery (2017 version)IBM MQ High Availabillity and Disaster Recovery (2017 version)
IBM MQ High Availabillity and Disaster Recovery (2017 version)
MarkTaylorIBM
 
IBM MQ in Containers - Think 2018
IBM MQ in Containers - Think 2018IBM MQ in Containers - Think 2018
IBM MQ in Containers - Think 2018
Robert Parker
 
IBM MQ cloud architecture blueprint
IBM MQ cloud architecture blueprintIBM MQ cloud architecture blueprint
IBM MQ cloud architecture blueprint
Matt Roberts
 
Hello, kafka! (an introduction to apache kafka)
Hello, kafka! (an introduction to apache kafka)Hello, kafka! (an introduction to apache kafka)
Hello, kafka! (an introduction to apache kafka)
Timothy Spann
 
MQ Guide France - IBM MQ and Containers
MQ Guide France - IBM MQ and ContainersMQ Guide France - IBM MQ and Containers
MQ Guide France - IBM MQ and Containers
Robert Parker
 
Websphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsWebsphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentals
Biju Nair
 
Apache Kafka Best Practices
Apache Kafka Best PracticesApache Kafka Best Practices
Apache Kafka Best Practices
DataWorks Summit/Hadoop Summit
 
Fundamentals of Apache Kafka
Fundamentals of Apache KafkaFundamentals of Apache Kafka
Fundamentals of Apache Kafka
Chhavi Parasher
 
Disaster Recovery and High Availability with Kafka, SRM and MM2
Disaster Recovery and High Availability with Kafka, SRM and MM2Disaster Recovery and High Availability with Kafka, SRM and MM2
Disaster Recovery and High Availability with Kafka, SRM and MM2
Abdelkrim Hadjidj
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
Jeff Holoman
 
IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction
ejlp12
 
IBM MQ on cloud and containers
IBM MQ on cloud and containersIBM MQ on cloud and containers
IBM MQ on cloud and containers
Robert Parker
 
An Introduction to the Message Queuing Technology & IBM WebSphere MQ
An Introduction to the Message Queuing Technology & IBM WebSphere MQAn Introduction to the Message Queuing Technology & IBM WebSphere MQ
An Introduction to the Message Queuing Technology & IBM WebSphere MQ
Ravi Yogesh
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
Saroj Panyasrivanit
 
IBM Message Hub service in Bluemix - Apache Kafka in a public cloud
IBM Message Hub service in Bluemix - Apache Kafka in a public cloudIBM Message Hub service in Bluemix - Apache Kafka in a public cloud
IBM Message Hub service in Bluemix - Apache Kafka in a public cloud
Andrew Schofield
 
IBM MQ Advanced - IBM InterConnect 2016
IBM MQ Advanced - IBM InterConnect 2016IBM MQ Advanced - IBM InterConnect 2016
IBM MQ Advanced - IBM InterConnect 2016
Leif Davidsen
 

Contenu connexe

Tendances

Apache Kafka Best Practices
Apache Kafka Best PracticesApache Kafka Best Practices
Apache Kafka Best Practices
DataWorks Summit/Hadoop Summit
 

Tendances (20)

Fault tolerant and scalable ibm mq
Fault tolerant and scalable ibm mqFault tolerant and scalable ibm mq
Fault tolerant and scalable ibm mq
 
Mq presentation
Mq presentationMq presentation
Mq presentation
 
IBM MQ: Using Publish/Subscribe in an MQ Network
IBM MQ: Using Publish/Subscribe in an MQ NetworkIBM MQ: Using Publish/Subscribe in an MQ Network
IBM MQ: Using Publish/Subscribe in an MQ Network
 
Kafka with IBM Event Streams - Technical Presentation
Kafka with IBM Event Streams - Technical PresentationKafka with IBM Event Streams - Technical Presentation
Kafka with IBM Event Streams - Technical Presentation
 
Whats new in MQ V9.1
Whats new in MQ V9.1Whats new in MQ V9.1
Whats new in MQ V9.1
 
IBM MQ Overview (IBM Message Queue)
IBM MQ Overview (IBM Message Queue)IBM MQ Overview (IBM Message Queue)
IBM MQ Overview (IBM Message Queue)
 
IBM MQ High Availabillity and Disaster Recovery (2017 version)
IBM MQ High Availabillity and Disaster Recovery (2017 version)IBM MQ High Availabillity and Disaster Recovery (2017 version)
IBM MQ High Availabillity and Disaster Recovery (2017 version)
 
IBM MQ in Containers - Think 2018
IBM MQ in Containers - Think 2018IBM MQ in Containers - Think 2018
IBM MQ in Containers - Think 2018
 
IBM MQ cloud architecture blueprint
IBM MQ cloud architecture blueprintIBM MQ cloud architecture blueprint
IBM MQ cloud architecture blueprint
 
Hello, kafka! (an introduction to apache kafka)
Hello, kafka! (an introduction to apache kafka)Hello, kafka! (an introduction to apache kafka)
Hello, kafka! (an introduction to apache kafka)
 
MQ Guide France - IBM MQ and Containers
MQ Guide France - IBM MQ and ContainersMQ Guide France - IBM MQ and Containers
MQ Guide France - IBM MQ and Containers
 
Websphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsWebsphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentals
 
Apache Kafka Best Practices
Apache Kafka Best PracticesApache Kafka Best Practices
Apache Kafka Best Practices
 
Fundamentals of Apache Kafka
Fundamentals of Apache KafkaFundamentals of Apache Kafka
Fundamentals of Apache Kafka
 
Disaster Recovery and High Availability with Kafka, SRM and MM2
Disaster Recovery and High Availability with Kafka, SRM and MM2Disaster Recovery and High Availability with Kafka, SRM and MM2
Disaster Recovery and High Availability with Kafka, SRM and MM2
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction
 
IBM MQ on cloud and containers
IBM MQ on cloud and containersIBM MQ on cloud and containers
IBM MQ on cloud and containers
 
An Introduction to the Message Queuing Technology & IBM WebSphere MQ
An Introduction to the Message Queuing Technology & IBM WebSphere MQAn Introduction to the Message Queuing Technology & IBM WebSphere MQ
An Introduction to the Message Queuing Technology & IBM WebSphere MQ
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 

Similaire à IBM MQ and Kafka, what is the difference?

Similaire à IBM MQ and Kafka, what is the difference? (20)

IBM Message Hub service in Bluemix - Apache Kafka in a public cloud
IBM Message Hub service in Bluemix - Apache Kafka in a public cloudIBM Message Hub service in Bluemix - Apache Kafka in a public cloud
IBM Message Hub service in Bluemix - Apache Kafka in a public cloud
 
IBM MQ Advanced - IBM InterConnect 2016
IBM MQ Advanced - IBM InterConnect 2016IBM MQ Advanced - IBM InterConnect 2016
IBM MQ Advanced - IBM InterConnect 2016
 
Velocity Conference '13: Asynchronous messaging for performance optimization,...
Velocity Conference '13: Asynchronous messaging for performance optimization,...Velocity Conference '13: Asynchronous messaging for performance optimization,...
Velocity Conference '13: Asynchronous messaging for performance optimization,...
 
CTU 2017 - I168 IBM MQ in the cloud
CTU 2017 - I168 IBM MQ in the cloudCTU 2017 - I168 IBM MQ in the cloud
CTU 2017 - I168 IBM MQ in the cloud
 
AWS Initiate Day Dublin 2019 – Breaking down the Monoliths
AWS Initiate Day Dublin 2019 – Breaking down the MonolithsAWS Initiate Day Dublin 2019 – Breaking down the Monoliths
AWS Initiate Day Dublin 2019 – Breaking down the Monoliths
 
GWC : MQ Light - from monolith to Microservices for speed and scale
GWC : MQ Light - from monolith to Microservices for speed and scaleGWC : MQ Light - from monolith to Microservices for speed and scale
GWC : MQ Light - from monolith to Microservices for speed and scale
 
Mq light, mq, and bluemix web sphere user group july 2015
Mq light, mq, and bluemix web sphere user group july 2015Mq light, mq, and bluemix web sphere user group july 2015
Mq light, mq, and bluemix web sphere user group july 2015
 
What's new in IBM MQ, March 2018
What's new in IBM MQ, March 2018What's new in IBM MQ, March 2018
What's new in IBM MQ, March 2018
 
WSO2 Message Broker - Product Overview
WSO2 Message Broker - Product OverviewWSO2 Message Broker - Product Overview
WSO2 Message Broker - Product Overview
 
IBM MQ V9 Overview
IBM MQ V9 OverviewIBM MQ V9 Overview
IBM MQ V9 Overview
 
IBM MQ Update, including 9.1.2 CD
IBM MQ Update, including 9.1.2 CDIBM MQ Update, including 9.1.2 CD
IBM MQ Update, including 9.1.2 CD
 
MQ Light for WTU
MQ Light for WTU MQ Light for WTU
MQ Light for WTU
 
Messaging in the Cloud with IBM MQ Light and IBM Bluemix
Messaging in the Cloud with IBM MQ Light and IBM BluemixMessaging in the Cloud with IBM MQ Light and IBM Bluemix
Messaging in the Cloud with IBM MQ Light and IBM Bluemix
 
What's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OSWhat's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OS
 
Connecting IBM MessageSight to the Enterprise
Connecting IBM MessageSight to the EnterpriseConnecting IBM MessageSight to the Enterprise
Connecting IBM MessageSight to the Enterprise
 
Event driven microservices
Event driven microservicesEvent driven microservices
Event driven microservices
 
AWS Initiate Day Manchester 2019 – AWS Breaking Down the Monoliths
AWS Initiate Day Manchester 2019 – AWS Breaking Down the MonolithsAWS Initiate Day Manchester 2019 – AWS Breaking Down the Monoliths
AWS Initiate Day Manchester 2019 – AWS Breaking Down the Monoliths
 
Understanding mq deployment choices and use cases
Understanding mq deployment choices and use casesUnderstanding mq deployment choices and use cases
Understanding mq deployment choices and use cases
 
Multi-cloud deployment with IBM MQ
Multi-cloud deployment with IBM MQMulti-cloud deployment with IBM MQ
Multi-cloud deployment with IBM MQ
 
MQTT Introduction
MQTT IntroductionMQTT Introduction
MQTT Introduction
 

Plus de David Ware

InterConnect 2016: What's new in IBM MQ
InterConnect 2016: What's new in IBM MQInterConnect 2016: What's new in IBM MQ
InterConnect 2016: What's new in IBM MQ
David Ware
 

Plus de David Ware (9)

IBM MQ What's new - Sept 2022
IBM MQ What's new - Sept 2022IBM MQ What's new - Sept 2022
IBM MQ What's new - Sept 2022
 
Designing IBM MQ deployments for the cloud generation
Designing IBM MQ deployments for the cloud generationDesigning IBM MQ deployments for the cloud generation
Designing IBM MQ deployments for the cloud generation
 
Whats new in IBM MQ; V9 LTS, V9.0.1 CD and V9.0.2 CD
Whats new in IBM MQ; V9 LTS, V9.0.1 CD and V9.0.2 CDWhats new in IBM MQ; V9 LTS, V9.0.1 CD and V9.0.2 CD
Whats new in IBM MQ; V9 LTS, V9.0.1 CD and V9.0.2 CD
 
InterConnect 2016: IBM MQ self-service and as-a-service
InterConnect 2016: IBM MQ self-service and as-a-serviceInterConnect 2016: IBM MQ self-service and as-a-service
InterConnect 2016: IBM MQ self-service and as-a-service
 
InterConnect 2016: What's new in IBM MQ
InterConnect 2016: What's new in IBM MQInterConnect 2016: What's new in IBM MQ
InterConnect 2016: What's new in IBM MQ
 
IBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM MQ: Managing Workloads, Scaling and Availability with MQ ClustersIBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
 
IBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
IBM MQ: An Introduction to Using and Developing with MQ Publish/SubscribeIBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
IBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
 
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ ClustersIBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
 
IBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
IBM WebSphere MQ: Using Publish/Subscribe in an MQ NetworkIBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
IBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
 

Dernier

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Steffen Staab
 
AI & Machine Learning Presentation Template
AI & Machine Learning Presentation TemplateAI & Machine Learning Presentation Template
AI & Machine Learning Presentation Template
Presentation.STUDIO
 
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionIntroducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
OnePlan Solutions
 
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM TechniquesAI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
VictorSzoltysek
 
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdfintroduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
VishalKumarJha10
 
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
Nitya salvi
 

Dernier (20)

%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
 
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language ModelsUnlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Models
 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand
 
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension AidDirect Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
 
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
 
AI & Machine Learning Presentation Template
AI & Machine Learning Presentation TemplateAI & Machine Learning Presentation Template
AI & Machine Learning Presentation Template
 
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionIntroducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
 
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM TechniquesAI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
 
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdfintroduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
 
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdfPayment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
 
Exploring the Best Video Editing App.pdf
Exploring the Best Video Editing App.pdfExploring the Best Video Editing App.pdf
Exploring the Best Video Editing App.pdf
 
Pharm-D Biostatistics and Research methodology
Pharm-D Biostatistics and Research methodologyPharm-D Biostatistics and Research methodology
Pharm-D Biostatistics and Research methodology
 
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
%in kaalfontein+277-882-255-28 abortion pills for sale in kaalfontein
 
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsRight Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
 
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park %in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
 
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
 
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerHow To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
 

IBM MQ and Kafka, what is the difference?

  • 1. TechCon 2020 IBM MQ and Kafka, what is the difference? David Ware IBM MQ Chief Architect
  • 2. Messaging is essential for building fully connected, efficient and scalable solutions. More now than ever before Messaging covers a broad spectrum of requirements
  • 3. Messaging Scalability Security Management High Availability There are many capabilities essential to every enterprise messaging solution IBM MQ and Kafka have both repeatedly been proven to tick these boxes…
  • 4. © 2020 IBM Corporation So just pick one and move on?
  • 5. © 2020 IBM Corporation So just pick one and move on? Perhaps, if you think everything looks like a nail
  • 6. © 2020 IBM Corporation So just pick one and move on? Perhaps, if you think everything looks like a nail Otherwise, let’s focus on what you’re trying to achieve…
  • 7. Critical data exchange: work that needs to be done Critical applications demand assured asynchronous interactions Essential capabilities: End-to-end once and once only Deliveryü Fine grain messaging Messages typically represent commands, queries and operations The message is a way to pass control from the originator of the message to the consumer
  • 8. Event Driven: building scalable microservices Microservices increases the need for communication. API-based interactions can build fragile and unscalable tight bonds between components Topics and Subscriptions Publishing and subscribing to events relaxes the coupling of microservices Events are messages that communicate that something has occurred Essential capabilities:
  • 9. Event Streaming: the expanding need for messaging Event Streaming brings data together from disparate sources, enabling even more responsive and engaging experiences for a wider set of users Scalable Subscription Stream History Efficient cloud and analytics applications utilize local decoupled buffers of event data Essential capabilities:
  • 10. The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
  • 11. © 2020 IBM Corporation To choose the right tool you’re probably going to need to go to the next level…
  • 12. © 2020 IBM Corporation Message consumption
  • 13. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption
  • 14. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
  • 15. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7
  • 16. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
  • 17. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
  • 18. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 … • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
  • 19. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
  • 20. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
  • 21. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER 0 1 23 45 67
  • 22. © 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots
  • 23. © 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots Kafka topic partitions…
  • 24. © 2020 IBM Corporation Queues and topics
  • 25. © 2019 IBM Corporation Comparing queues and topics QUEUE 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER PRODUCER PRODUCER 0 1 23 45 6 7 TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER CONSUMER 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Queues Topics • Each messages goes to a single consumer from the queue • Each message goes to every subscriber to the topic
  • 26. © 2019 IBM Corporation CONSUMERCONSUMER MQ and Kafka topics TOPIC PARTITION 0 1 2 3 4 5 6 7 CONSUMER GROUP CONSUMER GROUP PRODUCER PRODUCER CONSUMER CONSUMER TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER 1 2 3 4 5 6 70 1 2 3 4 5 6 70 IBM MQ Kafka
  • 27. © 2019 IBM Corporation Queues and topics IBM MQ Kafka Queues Primary MQ concept Queues are administratively or dynamically created Queues can be partitioned for scale All messages are stored on queues, even publications n/a Topics Core MQ capability Topics are dynamically created as needed by applications Each subscribing application has access to a dedicated queue of to-be-consumed messages matching their subscription. Primary Kafka concept Rolling history of messages maintained by Kafka. Topics can be partitioned for scale… Consumer’s offset within a shared stream of messages is maintained by Kafka.
  • 28. © 2020 IBM Corporation Scaling
  • 29. Horizontal scaling – workload balance the messages PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER IBM MQ Kafka PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER
  • 30. Kafka Clusters • A Kafka system is typically made up of multiple active nodes • Topics can be partitioned to enable workload balancing across those nodes, improving scale and availability of the topics • Kafka workload balances messages across partitions, either randomly or based on application provided context to enable per-partition ordering • Each topic partition is active in one broker at a time • The messages held in a topic partition can be replicated to other brokers to ensure data integrity across failures and automatic recovery
  • 31. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 Kafka partitions and consumer groups 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
  • 32. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
  • 33. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
  • 34. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups - p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers X
  • 35. IBM MQ Clusters and workload balancing PRODUCE MESSAGES TO QUEUE A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER • A single running instance of MQ is a queue manager. • These can act independently or collectively to build horizontally scaled solutions. • An MQ Cluster is a dynamically provisioned set of queue managers with the ability to partition queues across them. • MQ will workload balance inbound messages from producing applications across each partition of the queue. • With MQ’s Uniform Cluster capability, MQ will workload balance multiple consuming applications across each queue manager, ensuring messages are consumed efficiently. • MQ does not provide simple message ordering in these topologies • MQ’s HA technologies provide data integrity across failures and automatic recovery
  • 36. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances
  • 37. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20, … 01,11, … X
  • 38. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20 01,11 03,13,23
  • 39. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20,40 01,11 03,13,23,43 CONSUMER CONSUMER CONSUMER CONSUMER 30,50,70 32,52 31 21
  • 40. © 2019 IBM Corporation Scaling IBM MQ Kafka Consuming application is the bottleneck Scale out the consuming application Messaging system is the bottleneck Scale out the messaging system Increase the number of consuming applications from the existing queue/subscription. Increase the number of topic partitions to support a greater number of consuming applications in a consumer group. Increase the number of queue managers deployed and workload balance messages and consumers across them. Increase the number of Kafka broker nodes deployed and redistribute the topic partitions across them. †Good practice is to capacity plan for highest load and pre-define multiple topic partitions to support more consumers than you currently require, enabling more dynamic consumer scaling https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ † † † Unlike MQ, where new queue managers do not require copies of the existing messages, Kafka requires event histories to be copied to a new broker before it can be effective, slowing down dynamic scaling https://www.infoq.com/presentations/cloud-native-kafka-netflix/ ††
  • 41. © 2020 IBM Corporation Conversational messaging
  • 42. © 2019 IBM Corporation ConversationConversation ConversationConversation Fine grain communication? • MQ is designed for tens of thousands of lightweight queues and hundreds of thousands of dynamic topic strings per queue manager • This enables each conversation (e.g. reply flow) to be a dedicated queue or topic • Each consumer only sees its intended message • Dynamic create/deletion of queues and topic strings IBM MQ Equivalent Kafka model … • Kafka is designed for less, larger, configured topic partitions per broker • Recommendations are typically for 2-4k topic partitions per broker* • Consumers process every message in the topic from a point forward • This model is considered a Kafka anti-pattern. Recommendation is to rearchitect application to a stream processing pattern if possible** • https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions ** https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka PRODUCERPRODUCER PRODUCERPRODUCER CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … TOPIC 2 30 1 … …
  • 43. © 2020 IBM Corporation Message delivery
  • 44. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Any problem in the messaging system or the application may result in messages being lost Messages are not lost in the event of a failure, but any message may be duplicated Messages are not lost in the event of a failure and each message can be processed individually to prevent possible duplication An application can coordinate the processing of a message with a specific update to an external resource, such as a database
  • 45. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications
  • 46. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications Ideal for event streaming use case Ideal for critical data exchange use case
  • 47. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Higher levels of message reliability typically reduce the need for applications to consider error scenarios. However, higher message reliability typically reduces overall message performance and can limit overall availability The trade off Event streaming criticaldata exchange Reliability Performance
  • 48. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Kafka IBM MQ https://medium.com/@andrew_schofield/does-apache-kafka-do-acid-transactions-647b207f3d0e Sweet spots Default behavior. Some messages may be lost following an application or Kafka failure, but others persisted Default behavior. All messages of this type lost following an MQ failure by design Kafka well designed for this, producers can be configured to retry failed sends and consumers typically periodically update their offset Possible to design an application to behave like this through message batching and protocol optimizations, but not a key MQ scenario Possible within a closed Kafka system, but much harder with external systems. Technically possible to lock Kafka down to confirm every message for individual acknowledgment but loses benefits A natural fit for the destructive consume approach of MQ coupled with its core persistence model Supports Kafka-only transactions for exactly-once stream processing applications, but no external resource coordination A core MQ capability is as a resource manager (or coordinator) for global transactions Event streaming criticaldata exchange
  • 49. © 2020 IBM Corporation Message availability
  • 50. © 2019 IBM Corporation Per-message availability IBM MQ Kafka Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes. PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9
  • 51. © 2019 IBM Corporation Per-message availability IBM MQ Kafka PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 active/active partitioning of queues and topics, each partition is active/passive Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes.
  • 52. © 2020 IBM Corporation Summary
  • 53. © 2020 IBM Corporation If you want to build the best solution, you’re probably going to need more than a hammer
  • 54. The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
  • 55. Thank you David Ware IBM MQ Chief Architect — www.linkedin.com/in/dware1 © Copyright IBM Corporation 2020. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and ibm.com are trademarks of IBM Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available at Copyright and trademark information. 57TechCon 2020 © 2020 IBM Corporation
  • 56. TechCon 2020 / © 2020 IBM Corporation 58