When moving to the cloud you want to ensure that the deployment and management of your cloud queue managers is as easy and streamlined as possible. In this session we will look at a few tools you can use to deploy and manage your queue managers, as well as where you can find examples of these tools in action.
This presentation was given at the WebSphere User Group in Hursley, June 2017.
Real-time Tracking and Monitoring with Cargo Cloud Solutions.pptx
Deploying and managing IBM MQ in the Cloud
1. Rob Parker
IBM MQ Development
1 7/10/2017
Deploying and Managing
IBM MQ in the cloud
2. • Deploying and Orchestration
• Managing your Queue Manager In the cloud
• Monitoring your Queue Manager In the cloud
• Questions
Agenda
3. On-prem
… IBM MQ Appliance
AWS
AWS
AWS
Cloud
Message Hub
IBM Bluemix
(including Softlayer)
Distributed platforms
…
Private cloud
Run MQ, exactly how and where you need it
8. 9
Manage and script software installation
Similar to Chef, use in conjuction with an MQ install
package to automate and repeat installations
Provides an MQ cookbook to include in Chef recipes
Use to perform repeatable & automatable MQ
installations
Create immutable OS images with MQ pre-installed
Cookie-cutter installations of e.g. MQ & Redhat
Deploy images to chosen IaaS service
Installing MQ
https://developer.ibm.com/messaging/mq-on-cloud/
9. 10
Simple shell script can be a useful lowest
common denominator
Use in a heat template, docker image, VM…
Not dependent on specific framework
As well as being able to install MQ, can create
and start queue managers
Can combine with installation steps to give a
complete MQ environment
Creating & starting QMs
https://developer.ibm.com/messaging/mq-on-cloud/
10. 11
Deleting resources when they’re no longer needed
Orchestration isn’t limited to creating resources
Tear down resources when they’re no longer required
Quickly spin up everything necessary to run a scenario, tear everything down when it’s no longer
needed
Useful for running one-off tests, creating short-lived development environments
Keeps cloud costs to a minimum – only pay for what you need at the time
Don’t need to manually keep track of what you created
Repeatability
terraform apply
terraform destroy
https://developer.ibm.com/messaging/mq-on-cloud/
11. 12
Using tools to create a full QM
If you use Packer to create an image
You can use scripts to provision, deploy that image and start/configure the QM
https://developer.ibm.com/messaging/mq-on-cloud/
12. • MQ 8.0.0.4+ supported to run inside a Docker image
•Details: https://ibm.biz/mqdocker
• Brings the benefits of Docker to MQ
•Lightweight containers for running MQ
•Predictable and standardized units for deploying MQ
•Process, resource and dependency isolation
Binary image in Docker Hub Source in GitHub
Sample MQ Docker Container – Install and basic QM setup in 1 command
“docker run –e LICENSE=accept –e MQ_QMGR_NAME=QM1 ibmcom/mq”
Downloads MQ docker
image from DockerHub
Starts MQ docker
container
Creates
QM1
Starts
QM1
Runs optional
mqsc config script
1 2 3 4 5
13. MQ Advanced for Developers
in the Bluemix Container Service
It is now possible to spin up MQ queue managers in
the Bluemix Container Service with a couple of
clicks
The fastest way to get up and running with MQ for
development and experimentation
Pre-configured defaults mean instant access for
administration and messaging applications
15. MQ
MQ
MQ
Auto-scaling
MQ
x n
• Key benefit of cloud
technologies
• Change your infrastructure to
meet demand
• Pay for what you’re using
• Cookie-cutter model using
immutable software stacks
App
App
App
App
x n
16. AWS Auto-Scaling & Route53 DNS
EC2 auto-scaling group
MQ
EC2 Instance
MQ
EC2 Instance
MQ
EC2 Instance
MQ
EC2 Instance x n
• Manually or automatically scale the
number of EC2 instances
• Scaling policies define when instances
scale up or down
• Scheduled – e.g. scale up every
Monday morning, scale down Monday
evening
• Based on metrics - e.g. when queue
depths stay above a certain level for
15 minutes
• Have a listening app register new
instances in DNS with unique
hostname
AWS Service Discover Reference Architecture
https://aws.amazon.com/blogs/compute/service-discovery-
an-amazon-ecs-reference-architecture/
17. AWS Auto-Scaling
EC2 auto-scaling group
MQ
EC2 Instance
MQ
EC2 Instance
x 1
• Auto-scaling not just for very dynamic
systems
• Also appropriate for MQ failover
scenarios
• Auto-scaling group size set to 1
• Monitoring process checks the health
of the instance and replaces it if it has
died
Networked file system or block storage
(e.g. AWS EFS)
18. Kubernetes – Create clusters of MQ Containers
StatefulSet
mq0 mq1 mq2
mq0
data
mq1
data
mq2
data
ReplicaSet
Get
client
IP Service
19. Auto-Scaling Challenges
Challenges
• New queue managers need naming
• Permanently removing instances should only be done when
queues are empty
• Application queues
• Transmit queues
• Dead letter queue
• If you’re using clustering, samples like amqsclm can help
move your data to another queue manager before deleting it
• Clients need to be able to handle changing Queue
Managers
MQ(?)
EC2 Instance
Important
Messages
App
?
20. MQ Infrastructure
Service Discovery
• Runtime components understanding where they need to connect to
• MQ clients need to know where they can find a queue manager
• MQ queue managers need to know where they can find a queue
• MQ partial repositories need to know where the full repositories are
Registration
• Administrators & planners need to know what they’re running and where
• Out dated versions of MQ
• Installations that need patching
• Queue managers that are underutilised
• Queue managers that are close to capacity
Auto-Scaling Challenges
Admin
App
?
21. Constraints to bear in mind
• QM-scoped artefacts
• Durable subscriptions only exist where they were created
• Likewise for global transactions
• Performance
• Putting to the same QM as a getter is preferable to multiple
QM hops
• Availability
• A queue manager with no getters may have orphaned
messages
24. IBM Cloud Product Insights – New IBM Bluemix Service
• Supported in MQ 9.0.2+
• IBM Bluemix service that gives you an overview of your on-premise and cloud-hosted products
• See which queue managers (or Liberty/WAS servers, or IIB Brokers etc.) you have, which
versions they are, what platforms they’re on, how utilised they are.
https://developer.ibm.com/product-insights
V 9.0.2
26. IBM Cloud Product Insights
Products listed in one place to
drill down into
Individual queue managers
displayed, showing their
hostname, installation path, QM
name and QM ID
V 9.0.2
27. IBM Cloud Product Insights
Selecting a queue manager
displays information about
the queue manager
MQ version, install path,
platform etc.
V 9.0.2
28. IBM Cloud Product Insights
Selecting a queue
manager also displays
basic usage
information
Usage information
about the queue
manager (no. of put
messages & bytes)
V 9.0.2
29. IBM Cloud Product Insights
• Queue manager authorises itself using the
API key associated with the Product
Insights service
• Copy API key to file in /var/mqm (or
Windows equivalent)
• Update qm.ini with the
“BluemixRegistration” stanza
V 9.0.2
34. IBM Cloud Product Insights Log Management – Experimental Bluemix Service
• Based on Elasticsearch and
Kibana
• Pre-defined Kibana
dashboards show basic
overview of your MQ estate
• Customisable to show
different perspectives on
different dashboards
35. IBM Cloud Product Insights Log Management
• Help pane describes the
dashboard and how to use it
• Raw AMQERR messages for detailed investigation
• AMQERR activity over time, per queue manager
• Interactive – select a
queue manager in one
widget and the entire
dashboard updates
36. Centralised Monitoring – MQ $SYS topics
• First introduced on the MQ Appliance to aid remote monitoring & diagnosis
• Added to MQ distributed platforms in MQ V9
• Benefits over pre-V9 statistics messages
• They automatically get published when an app subscribes
• Metadata topics let you see what’s available to subscribe to
• e.g. $SYS/MQ/INFO/QMGR/MYQM/Monitor/METADATA/CLASSES
• New data available that wasn’t previously (e.g. DISK and CPU activity)
• Includes MQ activity trace capability
• e.g. $SYS/MQ/INFO/QMGR/MYQM/ActivityTrace/ChannelName/MY.SVRCONN
37. Centralised Monitoring – MQ $SYS topics
• Separated from the rest of the topic tree
• Root of the $SYS/MQ tree is
SYSTEM.ADMIN.TOPIC object
• No permissions inherited from objects
above SYSTEM.ADMIN.TOPIC
• Wildcard subscription at “#” doesn’t match
anything under $SYS/MQ/…
• Wildcard subscriptions under
$SYS/MQ/… are restricted
38. MQ MQ
MQ
MQ
MQ MQ
MQ
MQ
MQ MQ
MQ
MQ
Centralised Monitoring
MQ app consuming
$SYS/MQ messages
Managed subscriptions to
“$SYS/MQ/…” with remote
queue as destination
MQ
39. Centralised Monitoring – MQ $SYS topics
• Visualise queue depths, CPU & disk
spikes, # of client connections….
• See https://github.com/ibm-
messaging/mq-golang to try it for yourself
40. 41 7/10/2017
Prove it
“I can put and then get a message from a new
Docker container running MQ in a clean workspace in
about 5 mins… 1.5 mins with the image already
downloaded.”
--Robert Parker, June 2017
43. 44 7/10/2017
Notices and disclaimers
continued
Information concerning non-IBM products was obtained from the
suppliers of those products, their published announcements or other
publicly available sources. IBM has not tested those products in
connection with this publication and cannot confirm the accuracy of
performance, compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should be
addressed to the suppliers of those products. IBM does not warrant the
quality of any third-party products, or the ability of any such third-party
products to interoperate with IBM’s products. IBM expressly disclaims
all warranties, expressed or implied, including but not limited to, the
implied warranties of merchantability and fitness for a particular,
purpose.
The provision of the information contained herein is not intended to, and
does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS,
Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document
Management System™, FASP®, FileNet®, Global Business Services®,
Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®,
IBM Social Business®, Information on Demand, ILOG, Maximo®,
MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower,
PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®,
PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®,
PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®,
SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®,
Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and
System z® Z/OS, are trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. Other product
and service names might be trademarks of IBM or other companies. A
current list of IBM trademarks is available on the Web at "Copyright and
trademark information" at: www.ibm.com/legal/copytrade.shtml.