The Enterprise
Kafka Summit 2016
Rimas Silkaitis
how Heroku leveraged Kafka to
realize the original vision of the
enterprise service bus (esb)
Rimas Silkaitis
Product
Heroku Postgres Heroku Redis
What is an ESB?
Analyst Definitions
ESBs receive service consumption requests,
route the requests to the correct service
provider, transform the requests to a...
Uses a bus architecture for scalability and
reliability, establishes a unified communications
channel, provides integratio...
Vendor Definitions
Oracle Service Bus transforms complex and
brittle architectures into agile integration
networks by connecting, virtualizin...
Mule ESB is a lightweight Java-based
enterprise service bus (ESB) and integration
platform that allows developers to conne...
How should we define it?
¯_(ツ)_/¯
App
App
App
App
Finance
HR
BizOps
Inventory
ESB
Routing
Mediation
Messaging
Event
Processing
Service
Orchestration
ESB
Push
Centralized
Architecture
API
CLI GIT Dashboard
Data
Runtime
BizOps
$ heroku addons:create heroku-postgresql:standard-0 -a sushi
Creating postgresql-reticulated-74861... done, ($50.00/month)...
API
CLI GIT Dashboard
Data
Runtime
BizOps
API
CLI GIT Web
Data
Runtime
BizOps
THE API IS OUR ESB!
Problems Emerge
• Bottleneck in getting features released that touch API
• Difficult to get Analytics across the entire Org...
PUSH vs PULL
http://kafka.apache.org/documentation.html#design_pull
PUSH
PULL Producers Consumers
ESB Redefinition
• Pull over Push
• Consumers dictate their own consumption rate
• Independent development, scaling and sch...
CLI GIT Dashboard
API Data Runtime BizOps
Other Use Cases
Around Heroku
Operational Experience
aggregation
router
aggregation
router
metrics-ingest
route
r
metrics-ingest
route
r
ZK
postgres
metrics-ingest
router
aggr...
$ heroku kafka:list -a metaas-addons
=== Kafka Topics on KAFKA_URL
Name Messages Traffic
─────────────────────────────────...
What do we need to do
next?
Schema Registry
Security
as a Service
Creating kafka-lively-27896... done, (free)
Adding kafka-lively-27896 to sushi... done
Kafka cluster has been created and ...
Public Beta
https://heroku.com/kafka
Come See Us At Our Booth
We’re Hiring!
The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus, Rimas Silkaitis
The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus, Rimas Silkaitis
The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus, Rimas Silkaitis
Prochain SlideShare
Chargement dans…5
×

The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus, Rimas Silkaitis

2 979 vues

Publié le

The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus

Publié dans : Ingénierie

The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus, Rimas Silkaitis

  1. 1. The Enterprise Kafka Summit 2016 Rimas Silkaitis
  2. 2. how Heroku leveraged Kafka to realize the original vision of the enterprise service bus (esb)
  3. 3. Rimas Silkaitis Product
  4. 4. Heroku Postgres Heroku Redis
  5. 5. What is an ESB?
  6. 6. Analyst Definitions
  7. 7. ESBs receive service consumption requests, route the requests to the correct service provider, transform the requests to a format compatible with the service provider, wait for the results, and deliver them back to the service consumer. IDC http://www.idc.com/2010st/esbcm.html
  8. 8. Uses a bus architecture for scalability and reliability, establishes a unified communications channel, provides integration, mediation and communication services and is standards based. http://redmonk.com/public/OpenSourceESBs.pdf Redmonk
  9. 9. Vendor Definitions
  10. 10. Oracle Service Bus transforms complex and brittle architectures into agile integration networks by connecting, virtualizing, and managing interactions between services and applications. http://www.oracle.com/technetwork/middleware/service-bus/overview/index.html Oracle
  11. 11. Mule ESB is a lightweight Java-based enterprise service bus (ESB) and integration platform that allows developers to connect applications together quickly and easily, enabling them to exchange data. https://www.mulesoft.com/resources/esb/what-mule-esb Mulesoft
  12. 12. How should we define it?
  13. 13. ¯_(ツ)_/¯
  14. 14. App App App App Finance HR BizOps Inventory
  15. 15. ESB Routing Mediation Messaging Event Processing Service Orchestration
  16. 16. ESB Push Centralized
  17. 17. Architecture
  18. 18. API CLI GIT Dashboard Data Runtime BizOps
  19. 19. $ heroku addons:create heroku-postgresql:standard-0 -a sushi Creating postgresql-reticulated-74861... done, ($50.00/month) Adding postgresql-reticulated-74861 to neovintage... done Setting HEROKU_POSTGRESQL_GRAY_URL and restarting sushi... done, v249 The database should be available in 3-5 minutes. ! The database will be empty. If upgrading, you can transfer ! data from another database with pg:copy. Use `heroku pg:wait` to track status Use `heroku addons:docs heroku-postgresql` to view documentation.
  20. 20. API CLI GIT Dashboard Data Runtime BizOps
  21. 21. API CLI GIT Web Data Runtime BizOps THE API IS OUR ESB!
  22. 22. Problems Emerge • Bottleneck in getting features released that touch API • Difficult to get Analytics across the entire Org • Hiring has become difficult • API is down. So is the entire platform.
  23. 23. PUSH vs PULL http://kafka.apache.org/documentation.html#design_pull
  24. 24. PUSH
  25. 25. PULL Producers Consumers
  26. 26. ESB Redefinition • Pull over Push • Consumers dictate their own consumption rate • Independent development, scaling and scheduling • Stop the world processing doesn’t exist
  27. 27. CLI GIT Dashboard API Data Runtime BizOps
  28. 28. Other Use Cases Around Heroku
  29. 29. Operational Experience
  30. 30. aggregation router aggregation router metrics-ingest route r metrics-ingest route r ZK postgres metrics-ingest router aggregation router aggregation router aggregation router sink router aggregation router aggregation router metrics-api router
  31. 31. $ heroku kafka:list -a metaas-addons === Kafka Topics on KAFKA_URL Name Messages Traffic ─────────────────────────────────── ────────────── ───────────── ROUTER_REQUEST.v1 132,823.12/sec 14.6 mb/sec DYNO_MEMORY.v1 1,478.97/sec 187 kb/sec DYNO_LOAD.v1 1,468.01/sec 153 kb/sec ROUTER_STATUS_SUMMARY.v1 987.4/sec 28.3 kb/sec SYSLOG.v1 691.9/sec 76.8 kb/sec ROUTER_LATENCY_SUMMARY.v1 678.84/sec 37.5 kb/sec ROUTER_ERROR.v1 476.12/sec 93.9 kb/sec DYNO_ERROR.v1 354.06/sec 38.5 kb/sec DYNO_LOAD_SUMMARY.v1 241.33/sec 9.73 kb/sec DYNO_MEMORY_SUMMARY.v1 219.26/sec 25.1 kb/sec DYNO_ERROR_SUMMARY.v1 73.49/sec 1.83 kb/sec ROUTER_ERROR_SUMMARY.v1 12.35/sec 276 bytes/sec __consumer_offsets 4.1/sec 405 bytes/sec ROUTER_STATUS_PARTIAL_SUMMARY.v1 0.0/sec 0 bytes/sec ROUTER_LATENCY_PARTIAL_HISTOGRAM.v1 0.0/sec 0 bytes/sec ROUTER_REQUESTS.v1 0.0/sec 0 bytes/sec
  32. 32. What do we need to do next?
  33. 33. Schema Registry
  34. 34. Security
  35. 35. as a Service
  36. 36. Creating kafka-lively-27896... done, (free) Adding kafka-lively-27896 to sushi... done Kafka cluster has been created and will be available shortly Use `heroku addons:docs heroku-kafka` to view documentation. === KAFKA_URL Name: kafka-silhouetted-79119 Created: 2016-03-24 21:21 UTC Plan: Beta 3 Status: available Version: 0.9.0.0 Topics: 0 topics (see heroku kafka:list) Connections: 0 consumers (0 applications) Messages: 0.0 messages/sec Traffic: 0 Bytes/sec in / 0 Bytes/s out $ heroku kafka:info -a sushi $ heroku addons:create heroku-kafka:beta-3 -a sushi
  37. 37. Public Beta https://heroku.com/kafka
  38. 38. Come See Us At Our Booth We’re Hiring!

×