Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

10 Things Every Developer Using RabbitMQ Should Know

2 436 vues

Publié le

RabbitMQ is the most popular open-source message broker. It’s a de facto standard for message-based architectures. And yet, despite the abundant documentation and usage, developers and operators can still get tripped up on configuration and usage patterns.

Let’s face it: some of these best practices are hard to capture in docs. There’s a subtle difference between what RabbitMQ *can* do, and *how* you should use it in different scenarios. Now is your chance to hear from seasoned RabbitMQ whisperers, Jerry Kuch and Wayne Lund.

Join Pivotal’s Jerry, Senior Principal Software Engineer, and Wayne, Advisory Data Engineer, as they share their top ten RabbitMQ best practices. You’ll learn:

- How and when—and when *not*—to cluster RabbitMQ
- How to optimize resource consumption for better performance
- When and how to persist messages
- How to do performance testing
- And much more!

Jerry Kuch, Senior Principal Software Engineer
Wayne Lund, Advisory Data Engineer, Pivotal

Publié dans : Technologie
  • Soyez le premier à commenter

10 Things Every Developer Using RabbitMQ Should Know

  1. 1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Jerry Kuch - jkuch@pivotal.io Wayne Lund - wlund@pivotal.io 12 Dec 2018 10 Things Every Developer Using RabbitMQ Should Know
  2. 2. Introduction ● RabbitMQ is a complex, multi- faceted system ● RabbitMQ has many features ● A few basic ideas can go a long way in: ○ Understanding RabbitMQ ○ Using RabbitMQ effectively Broker Producer Message Exchange Bindings Queues Consumer
  3. 3. Agenda ●Ten Topics ●From Three Major Themes: ○ Guiding Principles ○ Messages and Their Handling ○ Resource Management ●Where to Go Next Broker Producer Message Exchange Bindings Queues Consumer
  4. 4. Guiding Principles Ten Best Practices
  5. 5. ● RabbitMQ is not a database ● Aim to keep queue lengths short in practice ○ Balancing ingress and egress rates is the first, best idea ○ Know what a “normal” length for a given queue is ● Problems with long queues: ○ Memory alarms, paging to disk, TCP back pressure ● Mind message sizes ● Monitor! Guiding Principles Item 1: “A Happy Rabbit is an Empty Rabbit"
  6. 6. ● RabbitMQ is written in Erlang ● Erlang concurrency is based on lightweight processes ○ Rabbit is built of a collection of these communicating processes ○ Each queue you create has its own queue process ○ Erlang processes are scheduled on a pool of OS threads ○ Queues are thus natural concurrency bottlenecks ● Mechanical sympathy for your system helps you better exploit it ● By contrast exchanges are not backed by their own processes Guiding Principles Item 2: “The Queue is Rabbit’s Basic Unit of Concurrency"
  7. 7. Messages and Their Handling Ten Best Practices
  8. 8. ● What makes us think anything happened in a distributed system? ● Transfer of Responsibility: ○ At any moment, know who is responsible for the existence and integrity of a message ○ Maintain the responsibility and hand it off in a disciplined way ● Consumer side: AMQP ACKs. ● Producer side: Publisher Confirms (also AMQP transactions) Messages and Their Handling Item 3: The “Transfer of Responsibility” Notion
  9. 9. ● Ways to consume: basic.get and basic.consume ● Most clients should favor basic.consume ● Prefetch (prefetch_count in basic.qos method): ○ How many messages a client is willing to take at a time ○ Broker will deliver up to that many and wait for ACK ● Small prefetches can inhibit throughput ● Large prefetch can result in one of many consumers on a queue monopolizing queue’s output ● Choose based on how fast consumer gets work done before ACKing Messages and Their Handling Item 4: Consumers Should Eat Not Nibble
  10. 10. ● Messages in many applications break into natural tiers. ● Many RabbitMQ features impose non-trivial costs: ○ Persistence ○ HA Mirroring of Queues ○ Clustering ○ Publisher Confirms and AMQP Transactions ○ Consumer ACKs vs noACK/autoACK ● Think about your traffic, the benefits of a feature and spend accordingly! Messages and Their Handling Item 5: Know Costs of Guarantees and Don’t Overpay Unnecessarily
  11. 11. Resource Management Ten Best Practices
  12. 12. ● Some objects are more heavyweight than others ● CPU and RAM are limitations more often than networking ● Management can impose overhead: ● Beware in large clusters of ‘detailed’ metrics ● Don’t hit the wall (and understand the padding in front of it)! ● Padding: Memory and disk alarms, TCP back pressure ● The Wall: Fatal resource exhaustion is fatal Resource Management Item 6: Don’t Overtax Your System’s Resources
  13. 13. ● Understand AMQP Connections vs. Channels ○ Connections are expensive ○ Channels try to amortize the expense via multiplexing ● Don’t churn connections and mind your OS limits ● Don’t share channels between threads ● Consider separate connections for publishers and consumers Resource Management Item 7: Manage Connections and Channels Thoughtfully
  14. 14. ● Auto-delete queues: delete when last subscriber unsubscribes. ● Exclusive queues: exclusive to one connection, deleted when connection closes. ● Queue length limit: as number of messages or count of bytes; ● Per-Queue Message TTL ● Per-Message TTL ● Queue TTL: whole queue vanishes after TTL if no consumers Resource Management Item 8: Tips for Cleaning Up Queues
  15. 15. ● Consider virtual hosts—remember they’re only namespaces. ● Beware noisy neighbors! ● Sometimes separate clusters can be best: ○ Keep noisy neighbors apart ○ Easier to debug, determine culpability, etc. Resource Management Item 9: Think About Isolation
  16. 16. ● Monitoring! Web UI, management HTTP API, logs ○ Workloads: https://github.com/rabbitmq/workloadsBuilt- in liveness check API ● Can wire API to Splunk, Nagios, etc. ● Monitor not just node health but also your message fabric: ○ Its shape and typical parameters ● Management statistics collection: collection rate configurable, beware heavy load ● Read, Learn, Experiment, Understand, Repeat. Resource Management Item 10: Know What’s Happening
  17. 17. Guiding Principles ● A Happy Rabbit is an Empty Rabbit ● The Queue is Rabbit’s Unit of Concurrency Messages and Their Handling ● The “Transfer of Responsibility” Notion ● Consumers Should Eat, Not Nibble ● Know Costs of Guarantees and Don’t Overpay Unnecessarily SUMMARY
  18. 18. Resource Management ● Don’t Overtax Your System’s Resources ● Manage Connections and Channels Thoughtfully ● Tips for Cleaning Up Queues ● Think About Isolation ● Know What’s Happening SUMMARY (Continued)
  19. 19. ● RabbitMQ Website: http://www.rabbitmq.com ● RabbitMQ in Action, Manning Publications ● RabbitMQ in Depth, Manning Publications ● Workloads: https://github.com/rabbitmq/workloads ● Marcial Rosales developer guide: https://github.com/MarcialRosales/rabbitmq-developer-guide ● Jack Vanlightly Blogs: https://jack-vanlightly.com/ ● CloudAMQP Best Practices: https://www.cloudamqp.com/blog/index.html Where to Learn More
  20. 20. Transforming How The World Builds Software © Copyright 2018 Pivotal Software, Inc. All rights Reserved.