Successfully reported this slideshow.
Introducing IBM Message Hub
Oliver Deakin
20/04/2016
Apache Kafka as a Service
To do…
 What is IBM Bluemix?
 What is Message Hub?
 What is Message Hub… really?
 What else have we done?
2
What is IBM Bluemix?
3
Bluemix is an open-standards, cloud-based platform for building,
running, and managing applications...
What is IBM Bluemix?
4
To do…
 What is IBM Bluemix?
 What is Message Hub?
 What is Message Hub… really?
 What else have we done?
5
What is Message Hub?
6
What is Message Hub?
7
To do…
 What is IBM Bluemix?
 What is Message Hub?
 What is Message Hub… really?
 What else have we done?
8
What is Message Hub… really?
 Kafka 0.9
 Some additions
– Multi-tenancy
– Security
– Confluent REST API
– Administration...
Message Hub is… Kafka
 Kafka 0.9
– Took early releases
– Built, tested and deployed
– Deployed as Docker containers
– Con...
Message Hub is… Kafka
11
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
Message Hub is… Multi-tenant
 Single Kafka cluster
 Support up to 500 tenants
 Restrictions on each tenant’s resources
...
Message Hub is… Multi-tenant
13
Kafka
Zookeeper
User1 App
Send to
mytopic
User2 App
Send to
mytopic
User1
client
Lookup Us...
Message Hub is… Secure
 TLS introduced in Kafka 0.9
 Clients authenticated with SASL Plain
 IBM cloud security standard...
Message Hub is… Secure
15
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
Message Hub is… Secure
16
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLS
Message Hub is… Secure
17
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLS
HA ...
Message Hub is… Secure
18
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLS
HA ...
Message Hub is… Secure
19
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
TLS
HA ...
Message Hub is… Secure
 Throughput of 900k msgs/s
– 100 byte messages
– 6 clients
– 1000 topics @1 partition
 Latency ra...
Message Hub is… Flexible
 Kafka REST Proxy
– Confluent platform 2.0
– Binary message format only
– No schema registry
21
Message Hub is… Flexible
22
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
Kafka
Zookeeper
User App
RESTR...
Message Hub is… Managed
 Kafka Administration REST interface
– Used to manage topic lifecycle
– Topics not auto-created
–...
Message Hub is… Managed
 Monitoring… and lots of it!
– collectd containers deployed on every host
– metrics pushed locall...
Message Hub is… Managed
25
To do…
 What is IBM Bluemix?
 What is Message Hub?
 What is Message Hub… really?
 What else have we done?
26
Message Connect
 Connect Enterprise and Cloud
– Experimental Bluemix service
– Messaging pipeline from MQ to Message Hub
...
Message Hub Incubator
 AMQP proxy layer
– Experimental Bluemix service
– Kafka is backing messaging engine
– Horizontally...
Bluemix Dedicated and Local
 Bluemix Dedicated
– IBM cloud and IBM managed
– Customer gets dedicated Bluemix environment
...
30
Twitter:
@IBMmessaging
DeveloperWorks:
https://developer.ibm.com/messaging/message-hub/
Bluemix blogs:
https://develope...
Prochain SlideShare
Chargement dans…5
×

Unified log-meetup-20160420

386 vues

Publié le

Message Hub - Apache Kafka as a Service - Unified Log Meetup talk

Publié dans : Logiciels
  • Soyez le premier à commenter

Unified log-meetup-20160420

  1. 1. Introducing IBM Message Hub Oliver Deakin 20/04/2016 Apache Kafka as a Service
  2. 2. To do…  What is IBM Bluemix?  What is Message Hub?  What is Message Hub… really?  What else have we done? 2
  3. 3. What is IBM Bluemix? 3 Bluemix is an open-standards, cloud-based platform for building, running, and managing applications Less managed More managed
  4. 4. What is IBM Bluemix? 4
  5. 5. To do…  What is IBM Bluemix?  What is Message Hub?  What is Message Hub… really?  What else have we done? 5
  6. 6. What is Message Hub? 6
  7. 7. What is Message Hub? 7
  8. 8. To do…  What is IBM Bluemix?  What is Message Hub?  What is Message Hub… really?  What else have we done? 8
  9. 9. What is Message Hub… really?  Kafka 0.9  Some additions – Multi-tenancy – Security – Confluent REST API – Administration REST API – Monitoring 9
  10. 10. Message Hub is… Kafka  Kafka 0.9 – Took early releases – Built, tested and deployed – Deployed as Docker containers – Continuous delivery pipeline  Community involvement – Active submitting bug fixes – KIP-43 – extend SASL mechanism support 10
  11. 11. Message Hub is… Kafka 11 Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper User App
  12. 12. Message Hub is… Multi-tenant  Single Kafka cluster  Support up to 500 tenants  Restrictions on each tenant’s resources – Up to 100 partitions per tenant (3-way replicated) – Max partition size 1G – Max message age 24 hours – Clean-up policy is deletion – Max message size 1M 12
  13. 13. Message Hub is… Multi-tenant 13 Kafka Zookeeper User1 App Send to mytopic User2 App Send to mytopic User1 client Lookup User1 tenant prefix, 1234 Add tenant prefix to mytopic Continue with topic name 1234mytopic User2 client Lookup User2 tenant prefix, ABCD Add tenant prefix to mytopic Continue with topic name ABCDmytopic
  14. 14. Message Hub is… Secure  TLS introduced in Kafka 0.9  Clients authenticated with SASL Plain  IBM cloud security standards – Customer message data encrypted everywhere – Data storage must be separated from public interface 14
  15. 15. Message Hub is… Secure 15 Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper User App
  16. 16. Message Hub is… Secure 16 Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper User App TLS
  17. 17. Message Hub is… Secure 17 Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper User App TLS HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy
  18. 18. Message Hub is… Secure 18 Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper User App TLS HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy TLS TLS TLS TLS TLS
  19. 19. Message Hub is… Secure 19 Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper User App TLS HA Proxy HA Proxy HA Proxy HA Proxy HA Proxy TLS TLS TLS TLS TLS
  20. 20. Message Hub is… Secure  Throughput of 900k msgs/s – 100 byte messages – 6 clients – 1000 topics @1 partition  Latency ranges from 10 – 70ms typically 20
  21. 21. Message Hub is… Flexible  Kafka REST Proxy – Confluent platform 2.0 – Binary message format only – No schema registry 21
  22. 22. Message Hub is… Flexible 22 Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper Kafka Zookeeper User App RESTREST HA Proxy HA Proxy HA Proxy HA Proxy HA ProxyHA Proxy
  23. 23. Message Hub is… Managed  Kafka Administration REST interface – Used to manage topic lifecycle – Topics not auto-created – Web UI also available for manual management  Topics can be created and deleted – Topics can have 1+ partitions – 100 partition limit per tenant across all topics 23
  24. 24. Message Hub is… Managed  Monitoring… and lots of it! – collectd containers deployed on every host – metrics pushed locally – forwarded to central graphite + grafana dashboards – continuous testing  Automated alerts – thresholds trigger support callouts – Slack integration to notify whole team 24
  25. 25. Message Hub is… Managed 25
  26. 26. To do…  What is IBM Bluemix?  What is Message Hub?  What is Message Hub… really?  What else have we done? 26
  27. 27. Message Connect  Connect Enterprise and Cloud – Experimental Bluemix service – Messaging pipeline from MQ to Message Hub – Bluemix apps have visibility of on-prem messages – Allows connectivity with cloud analytics and other services 27
  28. 28. Message Hub Incubator  AMQP proxy layer – Experimental Bluemix service – Kafka is backing messaging engine – Horizontally scalable  Target is to provide interoperability with Kafka clients 28
  29. 29. Bluemix Dedicated and Local  Bluemix Dedicated – IBM cloud and IBM managed – Customer gets dedicated Bluemix environment – Full Message Hub cluster – Spec defined by customer requirements – Available now  Bluemix Local – Customer’s cloud but IBM managed – TBD… 29
  30. 30. 30 Twitter: @IBMmessaging DeveloperWorks: https://developer.ibm.com/messaging/message-hub/ Bluemix blogs: https://developer.ibm.com/bluemix/blog/ LinkedIn: ibm.biz/ibmmessaging Message Hub service: https://console.ng.bluemix.net/catalog/services/message-hub

×