Message Hub - Apache Kafka as a Service - Unified Log Meetup talk

1. Introducing IBM Message Hub
Oliver Deakin
20/04/2016
Apache Kafka as a Service

2. To do…
 What is IBM Bluemix?
 What is Message Hub?
 What is Message Hub… really?
 What else have we done?
2

3. What is IBM Bluemix?
3
Bluemix is an open-standards, cloud-based platform for building,
running, and managing applications
Less managed More managed

4. What is IBM Bluemix?
4

5. To do…
 What is IBM Bluemix?
 What is Message Hub?
 What is Message Hub… really?
 What else have we done?
5

6. What is Message Hub?
6

7. What is Message Hub?
7

8. To do…
 What is IBM Bluemix?
 What is Message Hub?
 What is Message Hub… really?
 What else have we done?
8

9. What is Message Hub… really?
 Kafka 0.9
 Some additions
– Multi-tenancy
– Security
– Confluent REST API
– Administration REST API
– Monitoring
9

10. Message Hub is… Kafka
 Kafka 0.9
– Took early releases
– Built, tested and deployed
– Deployed as Docker containers
– Continuous delivery pipeline
 Community involvement
– Active submitting bug fixes
– KIP-43 – extend SASL mechanism support
10

11. Message Hub is… Kafka
11
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App

12. Message Hub is… Multi-tenant
 Single Kafka cluster
 Support up to 500 tenants
 Restrictions on each tenant’s resources
– Up to 100 partitions per tenant (3-way replicated)
– Max partition size 1G
– Max message age 24 hours
– Clean-up policy is deletion
– Max message size 1M
12

13. Message Hub is… Multi-tenant
13
Kafka
Zookeeper
User1 App
Send to
mytopic
User2 App
Send to
mytopic
User1
client
Lookup User1
tenant prefix,
1234
Add tenant
prefix to mytopic
Continue with topic
name 1234mytopic
User2
client
Lookup User2
tenant prefix,
ABCD
Add tenant
prefix to mytopic
Continue with topic
name ABCDmytopic

14. Message Hub is… Secure
 TLS introduced in Kafka 0.9
 Clients authenticated with SASL Plain
 IBM cloud security standards
– Customer message data encrypted everywhere
– Data storage must be separated from public interface
14

15. Message Hub is… Secure
15
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App

16. Message Hub is… Secure
16
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLS

17. Message Hub is… Secure
17
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLS
HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy

18. Message Hub is… Secure
18
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLS
HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy
TLS
TLS
TLS
TLS
TLS

19. Message Hub is… Secure
19
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLS
HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy
TLS
TLS
TLS
TLS
TLS

20. Message Hub is… Secure
 Throughput of 900k msgs/s
– 100 byte messages
– 6 clients
– 1000 topics @1 partition
 Latency ranges from 10 – 70ms typically
20

21. Message Hub is… Flexible
 Kafka REST Proxy
– Confluent platform 2.0
– Binary message format only
– No schema registry
21

22. Message Hub is… Flexible
22
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
RESTREST
HA Proxy HA Proxy HA Proxy HA Proxy HA ProxyHA Proxy

23. Message Hub is… Managed
 Kafka Administration REST interface
– Used to manage topic lifecycle
– Topics not auto-created
– Web UI also available for manual management
 Topics can be created and deleted
– Topics can have 1+ partitions
– 100 partition limit per tenant across all topics
23

24. Message Hub is… Managed
 Monitoring… and lots of it!
– collectd containers deployed on every host
– metrics pushed locally
– forwarded to central graphite + grafana dashboards
– continuous testing
 Automated alerts
– thresholds trigger support callouts
– Slack integration to notify whole team
24

25. Message Hub is… Managed
25

26. To do…
 What is IBM Bluemix?
 What is Message Hub?
 What is Message Hub… really?
 What else have we done?
26

27. Message Connect
 Connect Enterprise and Cloud
– Experimental Bluemix service
– Messaging pipeline from MQ to Message Hub
– Bluemix apps have visibility of on-prem messages
– Allows connectivity with cloud analytics and other services
27

28. Message Hub Incubator
 AMQP proxy layer
– Experimental Bluemix service
– Kafka is backing messaging engine
– Horizontally scalable
 Target is to provide interoperability with Kafka clients
28

29. Bluemix Dedicated and Local
 Bluemix Dedicated
– IBM cloud and IBM managed
– Customer gets dedicated Bluemix environment
– Full Message Hub cluster
– Spec defined by customer requirements
– Available now
 Bluemix Local
– Customer’s cloud but IBM managed
– TBD…
29

30. 30
Twitter:
@IBMmessaging
DeveloperWorks:
https://developer.ibm.com/messaging/message-hub/
Bluemix blogs:
https://developer.ibm.com/bluemix/blog/
LinkedIn:
ibm.biz/ibmmessaging
Message Hub service:
https://console.ng.bluemix.net/catalog/services/message-hub