Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Advanced Messaging with       Apache ActiveMQBosanac DejanMay 2011                                                        ...
About me Bosanac Dejan Senior Software Engineer at FUSESource - http://  fusesource.com Apache ActiveMQ committer and P...
What we are going to cover?       What is ActiveMQ       The Basics       Flow control       Scaling       High Avail...
What is ActiveMQ?                                                                                                         ...
Apache ActiveMQ Apache ActiveMQ     • Leading Open Source messaging platform     • Supported Java Standards:             ...
Background       ActiveMQ started in 2005 at CodeHaus       Moved to Apache Software Foundation in 2006       1,117,537...
The Basics                                                                                                                ...
Messaging is Loosely coupled exchange of messages between  applications Location transparency Can be persistent or non-...
Topics                                                                                                                 A P...
Topics One message goes to 0-to-many consumers  based on the current subscribers Think like mailing lists or discussion ...
Queues                                                                                                                  A ...
Queues    Messages are load balanced across many consumers    Each message goes to exactly one consumer    Consumers co...
Challanges Create a general messaging platform Support variety of use-cases     • Large number of clients     • Large nu...
Flow Control                                                                                                              ...
Flow Control - Why? Dealing with deep queues Dealing with slow consumers We want to prevent broker from being flooded wi...
Flow Control - How? Message Cursors Producer Flow Control                                                               ...
Flow Control - Cursors                                                                                                    ...
Flow Control - Store-based Cursor                                                                                         ...
Flow Control - Non-persistent cursor                                                                                      ...
Flow Control - Limits                        Per destination     <destinationPolicy>           <policyMap>             <po...
Flow Control - Producer Flow Control Throttling producer speed to the speed of consumers                                 ...
Flow Control - VM Cursor     <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">       <pendingQueuePolic...
Scaling                                                                                                                  A...
Scaling - Types Vertical scaling Horizontal scaling Traffic partitioning                                               ...
Vertical Scaling     Increase load capacity using a single broker can handle.     Problems:     • Thread count     • Memor...
Vertical Scaling - Threads     What are threads used for     • For Connections - Thread per Connection (blocking transport...
Vertical Scaling - Number of Connections     Use non-blocking transport     <transportConnectors>       <transportConnecto...
Vertical Scaling - Number of Destinations     Don’t use dedicated task runner      ACTIVEMQ_OPTS="-Dorg.apache.activemq.Us...
Vertical Scaling - Memory Give broker enough memory     ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRu...
Vertical Scaling - CPU Disable tight encoding It uses more CPU to create smaller packets uri = "failover://(tcp://localho...
Vertical Scaling - Conclusion     There is a limit to the scalability a single machine can                               g...
Horizontal Scaling     Increase load capacity using networked brokers  Concepts: Network of Broker                       ...
Horizontal Scaling - Network of Brokers                                                                                   ...
Horizontal Scaling - NOB Usage                                                       Configuration                <networkC...
Horizontal Scaling - Conclusion More latency in processing messages Beware of complex topologies                        ...
Hybrid Scaling     Partition traffic to more non-connected brokers                                                        ...
Hybrid Scaling - Conclusion Pros     • You can use all the tuning techniques used in Vertical scaling     • Have better H...
High Availability                                                                                                         ...
High Availability Pure Master/Slave JDBC Master/Slave Shared File System Master/Slave                                  ...
Pure Master-Slave                                                                                                         ...
Pure Master-Slave Shared nothing Fully replicated     • All messages     • All acknowledgements     • All transactions ...
JDBC Master-Slave                                                                                                         ...
JDBC Master-Slave    Extreme reliability – but not as fast    Recommended if already using an enterprise database    No...
Shared Storage Master-Slave                                                                                               ...
Shared Storage Master-Slave    Recommended if you have a SAN    No restriction on number of slaves    Simple configurati...
Future                                                                                                                  A ...
Future - ActiveMQ Apollo http://activemq.apache.org/apollo ActiveMQ 5.x reached scalability and performance  limits with...
Future - ActiveMQ Apollo    Reactor Based Thread Model    Scala 2.8 Implementation    Protocol Agnostic    REST Based ...
Future - ActiveMQ Apollo Performance                            http://hiramchirino.net/blog                              ...
Conclusions    Dynamic community    Leading in terms of messaging innovation    Built for Enterprise    Scalable, Good...
Questions? ActiveMQ Web sites:     • http://activemq.apache.org/     • http://fusesource.com/products/enterprise-activemq...
Prochain SlideShare
Chargement dans…5
×

Advanced messaging with Apache ActiveMQ

Features of ActiveMQ message broker for dealing with high load, scaling and high-availability

  • Soyez le premier à commenter

Advanced messaging with Apache ActiveMQ

  1. 1. Advanced Messaging with Apache ActiveMQBosanac DejanMay 2011 A Progress Software1 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  2. 2. About me Bosanac Dejan Senior Software Engineer at FUSESource - http:// fusesource.com Apache ActiveMQ committer and PMC member Co-author of ActiveMQ in Action A Progress Software 2 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  3. 3. What we are going to cover? What is ActiveMQ The Basics Flow control Scaling High Availability Future Conclusion A Progress Software 3 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  4. 4. What is ActiveMQ? A Progress Software4 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  5. 5. Apache ActiveMQ Apache ActiveMQ • Leading Open Source messaging platform • Supported Java Standards: o JMS 1.1, J2EE 1.4, JCA 1.5 and XA Reliable, high performance messaging • Out-performs many legacy proprietary message queues • Configurable for many different deployments Multi-Protocol/Multi-Language Support A Progress Software 5 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  6. 6. Background ActiveMQ started in 2005 at CodeHaus Moved to Apache Software Foundation in 2006 1,117,537 lines of code 24 committers Now the most widely used open source messaging system on the planet A Progress Software 6 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  7. 7. The Basics A Progress Software7 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  8. 8. Messaging is Loosely coupled exchange of messages between applications Location transparency Can be persistent or non-persistent Can be transactional A Progress Software 8 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  9. 9. Topics A Progress Software9 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  10. 10. Topics One message goes to 0-to-many consumers based on the current subscribers Think like mailing lists or discussion forums Ideal for publishing business events Distributed observer pattern Allows one part of your system to notify anyone else who may be interested in an event A Progress Software10 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  11. 11. Queues A Progress Software11 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  12. 12. Queues Messages are load balanced across many consumers Each message goes to exactly one consumer Consumers compete for messages Its easy to browse and monitor queues Ideal for grid style applications A Progress Software12 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  13. 13. Challanges Create a general messaging platform Support variety of use-cases • Large number of clients • Large number of destinations • Slow consumers Provide enterprise feaures • Security • High availability • Management • etc A Progress Software13 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  14. 14. Flow Control A Progress Software14 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  15. 15. Flow Control - Why? Dealing with deep queues Dealing with slow consumers We want to prevent broker from being flooded with messages We want to prevent broker running out of memory and other resources A Progress Software15 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  16. 16. Flow Control - How? Message Cursors Producer Flow Control A Progress Software16 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  17. 17. Flow Control - Cursors A Progress Software17 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  18. 18. Flow Control - Store-based Cursor A Progress Software18 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  19. 19. Flow Control - Non-persistent cursor A Progress Software19 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  20. 20. Flow Control - Limits Per destination <destinationPolicy> <policyMap> <policyEntries> <policyEntry queue=">" memoryLimit="10mb"/> </policyEntries> </policyMap> </destinationPolicy> System settings <systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="256 mb" /> </memoryUsage> <storeUsage> <storeUsage limit="100 gb" /> </storeUsage> <tempUsage> <tempUsage limit="10 gb" /> </tempUsage> </systemUsage> </systemUsage> A Progress Software20 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  21. 21. Flow Control - Producer Flow Control Throttling producer speed to the speed of consumers A Progress Software21 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  22. 22. Flow Control - VM Cursor <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"> <pendingQueuePolicy> <vmQueueCursor/> </pendingQueuePolicy> </policyEntry> A Progress Software22 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  23. 23. Scaling A Progress Software23 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  24. 24. Scaling - Types Vertical scaling Horizontal scaling Traffic partitioning A Progress Software24 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  25. 25. Vertical Scaling Increase load capacity using a single broker can handle. Problems: • Thread count • Memory usage • CPU usage A Progress Software25 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  26. 26. Vertical Scaling - Threads What are threads used for • For Connections - Thread per Connection (blocking transport) • For Dispatching - Thread per Destination A Progress Software26 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  27. 27. Vertical Scaling - Number of Connections Use non-blocking transport <transportConnectors> <transportConnector name="nio" uri="nio://0.0.0.0:61616"/> </<transportConnectors> Enables handling large number of clients A Progress Software27 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  28. 28. Vertical Scaling - Number of Destinations Don’t use dedicated task runner ACTIVEMQ_OPTS="-Dorg.apache.activemq.UseDedicatedTaskRunner=false" Use optimized dispatch for queues <destinationPolicy> <policyMap> <policyEntries> <policyEntry topic=">" optimizedDispatch="true"> ... </policyEntry> </policyEntries> </policyMap> </destinationPolicy> A Progress Software28 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  29. 29. Vertical Scaling - Memory Give broker enough memory ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRunner=false" Configure big enough memory usage <systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="1024 mb" /> </memoryUsage> ... </systemUsage> </systemUsage> A Progress Software29 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  30. 30. Vertical Scaling - CPU Disable tight encoding It uses more CPU to create smaller packets uri = "failover://(tcp://localhost:61616wireFormat.tightEncodingEnabled=false)"; A Progress Software 30 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  31. 31. Vertical Scaling - Conclusion There is a limit to the scalability a single machine can give A Progress Software31 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  32. 32. Horizontal Scaling Increase load capacity using networked brokers Concepts: Network of Broker A Progress Software32 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  33. 33. Horizontal Scaling - Network of Brokers A Progress Software33 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  34. 34. Horizontal Scaling - NOB Usage Configuration <networkConnector name="broker1-broker2" uri="static:(tcp://broker2:61617)" dynamicOnly="true" prefetchSize="1000" conduitSubscriptions="true" decreaseNetworkConsumerPriority="true" suppressDuplicateTopicSubscriptions="true" networkTTL="3"> </networkConnector> Connecting failover://(tcp://broker1:61616,tcp://broker2:61616)?randomize=true A Progress Software34 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  35. 35. Horizontal Scaling - Conclusion More latency in processing messages Beware of complex topologies A Progress Software35 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  36. 36. Hybrid Scaling Partition traffic to more non-connected brokers A Progress Software36 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  37. 37. Hybrid Scaling - Conclusion Pros • You can use all the tuning techniques used in Vertical scaling • Have better Horizontal scaleability than using Network Of Brokers (Less broker cross talk) Cons • Added complexity required on the end user Application A Progress Software37 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  38. 38. High Availability A Progress Software38 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  39. 39. High Availability Pure Master/Slave JDBC Master/Slave Shared File System Master/Slave A Progress Software39 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  40. 40. Pure Master-Slave A Progress Software40 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  41. 41. Pure Master-Slave Shared nothing Fully replicated • All messages • All acknowledgements • All transactions Slave does not start any transports or network connections A Progress Software41 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  42. 42. JDBC Master-Slave A Progress Software42 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  43. 43. JDBC Master-Slave Extreme reliability – but not as fast Recommended if already using an enterprise database No restriction on number of slaves Simple configuration A Progress Software43 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  44. 44. Shared Storage Master-Slave A Progress Software44 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  45. 45. Shared Storage Master-Slave Recommended if you have a SAN No restriction on number of slaves Simple configuration N.B. – ensure file locking works – and times out – NFSv4 good! A Progress Software45 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  46. 46. Future A Progress Software46 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  47. 47. Future - ActiveMQ Apollo http://activemq.apache.org/apollo ActiveMQ 5.x reached scalability and performance limits with the current architecture New broker core A Progress Software47 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  48. 48. Future - ActiveMQ Apollo Reactor Based Thread Model Scala 2.8 Implementation Protocol Agnostic REST Based Management A Progress Software48 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  49. 49. Future - ActiveMQ Apollo Performance http://hiramchirino.net/blog 10 producers/10 consumers single topic using Stomp 20 byte payload A Progress Software49 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  50. 50. Conclusions Dynamic community Leading in terms of messaging innovation Built for Enterprise Scalable, Good Performance, Reliable A Progress Software50 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  51. 51. Questions? ActiveMQ Web sites: • http://activemq.apache.org/ • http://fusesource.com/products/enterprise-activemq/ Blog: • http://www.nighttale.net/ Twitter: • http://twitter.com/dejanb • http://twitter.com/fusenews A Progress Software51 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company

    Soyez le premier à commenter

    Identifiez-vous pour voir les commentaires

  • AdeLucas1

    Nov. 18, 2015
  • moviesnmusic

    Dec. 16, 2015
  • ssuser52c883

    Feb. 22, 2016
  • hyoungseunglee

    Mar. 2, 2016
  • AlejandroFlores158

    Mar. 13, 2016
  • jesusvegagonzalez

    Apr. 5, 2016
  • PhaniTurlapati

    Apr. 14, 2016
  • pvmsikrsna

    May. 12, 2016
  • MuralimohanDintakurt

    Aug. 28, 2016
  • MerkerXu

    Sep. 14, 2016
  • AlexMart001

    Nov. 1, 2016
  • rafaelcba

    Dec. 30, 2016
  • sharmisthamona

    Feb. 16, 2017
  • TatuMikkonen

    Mar. 11, 2017
  • hyh961

    Apr. 15, 2017
  • DariuszFurmaniak

    May. 6, 2017
  • kunkunur

    Jun. 2, 2017
  • EminBAHADIR

    Sep. 17, 2017
  • AmarBabuKakumanu

    Oct. 12, 2017
  • jadeflon

    Oct. 24, 2017

Features of ActiveMQ message broker for dealing with high load, scaling and high-availability

Vues

Nombre de vues

40 492

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

5 203

Actions

Téléchargements

1 150

Partages

0

Commentaires

0

Mentions J'aime

68

×