Cloud-native approach of implementing message broker using KubeMQ in k8s cluster. This approach is fast from the traditional implementation (using Apache kafka, RabbitMq). This presentation was given in OSCONF 2020 supported by Collanix and Docker Bangalore Community.

1. Implementing Message Broker in Kubernetes

2. $whoami
 Suman Chakraborty - Senior Devops
Engineer @SAP Labs
 Community member & Speaker -
Docker Bangalore, CNCF Bangalore
group
 Tech Blogger on PaaS, Cloud-Native
& Microservices
https://www.linkedin.com/in/
schakraborty007/
@itsmesumanc

3. Agenda
 Need for a Message Broker ?
 The Solution - KubeMQ
 Messaging Patterns
 Kubernetes Messaging Use-Cases
 Demo
 KubeMQ vs Other Message Brokers

4. Need for a message broker ?
 Kubernetes has become the de-facto standard for building and managing
microservices, allowing seamless container management, scalability and service
extensibility.
 Managing a distributed architecture means processing high volume of traffic flow
of messages as each data model is disintegrated from the system.
 Problem arises, when the model grows (adding more data points, IoT hub,etc),
that impact the communication process across the system.
 Enterprises attempt to solve this communication gap in Kubernetes using a Point-
To-Pont connectivity system such as REST.
 However, REST is not effectively “cloud-native” in sense as its expensive, time-
consuming and requires additional maintenance.
 Building an effective cloud-native messaging broker system for k8s involves re-
architecting the stack and deploying a single focal point of communication for
better communication. This ensures that each service communicates with the
message queue broker in its own language

5. The Solution – KubeMQ
 Enterprise-grade message broker and message queue, scalable, high available and
secured
 Written in Golang, created as a small light-weight container (~ 30 MB)
 Can be deployed through Operator for full LM operation
 Supports both synchronous and asynchronous message relays – for ‘At Most Once
Delivery’ and ‘At Least Once Delivery models’
 Supports gRPC, Rest and WebSocket Transport protocols with TLS support
 Scales instantly and work with or without persistent volume
 Supported on all Kubernetes environments ( Vanilla installation, Rancher k3s, Microk8s
)
 Supports message masticating and smart routing, authentication and authorization
 No Message broker configuration needed (i.e., queues, exchanges)
 Supported development languages -> .Net, Java, Python, Go and NodeJS SDKs
 Observability support: Prometheus, Jeager, Zipkin, AWS X-Ray, Datadog, Google Stack,
Driver, HoneyComb

6. KubeMQ is part of CNCF landscape under Streaming and Messaging

7. Messaging Patterns

8. Kubernetes messaging use-cases
Multistage Pipeline :- This pattern processes the message between services in a coordinated
workflow.

9. Tasks Queue : - An A-synchronic pattern that has multiple producers<-> many consumers
relation to process a job. Each service is pulling messages from the queue to process it. The
sequence of processing the messages is not important as each worker takes the task from
the queue and process it in his own time.

10. Pub/Sub Stream: - This pattern streams data from many external sources (big data, IoT) and
process it in a dedicated service such as database, pipeline, machine learning, storage, etc.
Useful for multiple producers to small number of consumers relation.

11. Pub/Sub Real-time: - This A-Synchronic pattern is suitable for cases of a small number of
producers sending real-time data to many consumers. The message is sent to a channel,
where multiple services are subscribed to

12. Plans & Pricing

14. Advantages
 Robust messaging queue system
 Secured system
 Low DevOps maintenance system
 Well-connected ecosystem for logging in Kubernetes
 Rapid Deployment
 Configurable to handle change in cluster configuration

15. KubeMq vs other Message brokers
KubeMq Apache Kafka
Ultra-Simple & can be deployed by
any DevOps Member Level
Challenging, requires expert help
Lightweight and Small Footprint
Container (~90MB). No need for
additional components
Heavy and Resource-Demanding
(~2.1 GB) (three Kafka containers +
three ZooKeeper containers)
KubeMq is faster as its built on Go Kafka requires dedicated setup on
zookeeper, has a stateful setup
Effortless, developer-oriented, no
need to configure broker
components separately
Complex and dedicated setup
required (additional knowledge on
Java and Scala skillsets)
All messaging patterns are Only Async messaging patterns are
supported
Off-the-Shelf integrations, CNCF-
oriented
Requires additional configurations
and adjustments

16. References
 https://docs.kubemq.io/
 https://kubemq.io/blog/

17. THANK YOU