Building an Active-Active IBM MQ System

TechCon 2020
M04: Building an Active/Active IBM MQ
System
Matthew Whitehead
MQ Architect, Distributed Platforms
© 2020 IBM Corporation

Contents
Part One
• Different types of availability
• Recap benefits of HA queue managers
PartTwo
• Active/Active availability
• Uniform Clusters
TechCon 2020 / © 2020 IBM Corporation
See “M02 - Ensuring message high
availability with MQ systems” for
more detail

Fault Toleration
Queue Manager
Single
Queue Manager Queue Manager Queue Manager Queue Manager
Multiple
100%
0%
100%
0%
availability
availability
More is better for availability

Queue Manager Queue Manager Queue Manager
It’s not just the queue managers…
Client Client Client
Step 1
Horizontally scale the application into multiple instances, all
performing the same role
A queue manager works better when there are multiple
applications working in parallel Queue Manager
Step 2
Horizontally scale the queue managers
Create multiple queue managers with the ‘same’ configuration
Distribute the application instances across the queue managers
Client Client Client Client Client Client

Let’s go through that
availability thing one step at
a time…

Node
Single, non-HA queue manager
App App App
100%
0%
100%
0%
System
availability
Message
availability
Queue Manager

Node B Node C
Node A
Single HA queue manager
App App App
Queue Manager
Queue Manager
Highly available
queue manager
and queue
instances
100%
0%
100%
0%
System
availability
Message
availability
Queue Manager
Active/Standby

Node B Node C
Node A
Multiple HA queue managers
Queue Manager 1
Queue Manager 1
Highly available
queue manager
and queue
instances
Queue Manager 2
Queue Manager 2
Highly available
queue manager
and queue
instances
Queue Manager 3
Queue Manager 3
Highly available
queue manager
and queue
instances
App App App App
App
App App App
App
Queue Manager 1
Queue Manager 2
Queue Manager 3

Node B Node C
Node A
Multiple HA queue managers – coupled applications
Queue Manager 1
Queue Manager 1
Highly available
queue manager
and queue
instances
Queue Manager 2
Queue Manager 2
Highly available
queue manager
and queue
instances
Queue Manager 3
Queue Manager 3
Highly available
queue manager
and queue
instances
App App App App
App
App App App
App
Connect: QMgr1 Connect: QMgr2 Connect: QMgr3
100%
0%
100%
0%
System
availability
Message
availability
Queue Manager 2
Queue Manager 1
Queue Manager 3

Node B Node C
Node A
Multiple HA queue managers – coupled applications
Queue Manager 1
Queue Manager 1
Highly available
queue manager
and queue
instances
Queue Manager 2
Queue Manager 2
Highly available
queue manager
and queue
instances
Queue Manager 3
Queue Manager 3
Highly available
queue manager
and queue
instances
App App App App
App
App App App
App
Connect: QMgr1 Connect: QMgr2 Connect: QMgr3
Queue Manager 2
Queue Manager 1
Queue Manager 3
Active/Passive
100%
0%
100%
0%
System
availability
Message
availability

Node B Node C
Node A
Multiple HA queue managers – decoupled applications
Queue Manager 1
Queue Manager 1
Highly available
queue manager
and queue
instances
Queue Manager 2
Queue Manager 2
Highly available
queue manager
and queue
instances
Queue Manager 3
Queue Manager 3
Highly available
queue manager
and queue
instances
App App App App
App
App App App
App
Connect: QMgrGroup
100%
0%
100%
0%
System
availability
Message
availability
Queue Manager 1
Queue Manager 2
Queue Manager 3

Node B Node C
Node A
Multiple HA queue managers – decoupled applications
Queue Manager 1
Queue Manager 1
Highly available
queue manager
and queue
instances
Queue Manager 2
Queue Manager 2
Highly available
queue manager
and queue
instances
Queue Manager 3
Queue Manager 3
Highly available
queue manager
and queue
instances
App App App App
App
App App App
App
Connect: QMgrGroup
Queue Manager 1
Queue Manager 2
Queue Manager 3
Active/Active

Queue Manager
Active/passive
o Simple
o Invisible to applications
o Every outage is high impact
o Restart times can grow
o Limited by maximum system size
Active/active (+passive)
o Tolerate partial failures
o More suited to cloud deployment styles
o Enables rolling upgrades
o Reduced restart times – improved availability
o Unlimited by system size
o Visible to applications – limitations apply
o Potentially more complicated
Queue Manager
100%
0%
100%
0%
System
availabilit
y
Message
availabilit
y
100%
0%
100%
0%
System
availabilit
y
Message
availabilit
y

Message Availability:
Highly available queue managers

Message high availability
– Consider a single message
– Tied to a single runtime, on a single piece of
hardware
– Any failure locks it away until recovery
completes
The problem The objective
– Messages are not tied to a single anything
– In the event of a failure, there is a fast route
to access the message

– Messages are highly available, through
replication
– Only one runtime is the leader and has access
to the messages at a time
– A failure results in a new leader taking over
– Any message is available from any runtime at
any time
– Coordinated access to each message
– A failed runtime does not prevent access to a
message by another runtime
Active / active messages Active / passive messages

Active / active messages Active / passive messages
IBM MQ Distributed HA solutions
(but there’s nothing to stop you having multiple
different queue managers to share the load and the
risk.
More later)
IBM MQ for z/OS shared queues

IBM MQ Distributed HA solutions
MQ managed
The resilient data and the automatic takeover is
provided by the MQ system
Externally managed
External mechanisms are relied on to protect the
data and provide automatic takeover capabilities
System managed
HA
QMgr QMgr
Multi-instance queue
managers
QMgr QMgr
MQ Appliance
QMgr
QMgr
Replicated data
queue managers
QMgr QMgr QMgr

Service Availability:
Uniform Clusters

Moving from a single queue manager to multiple
queue managers:
• reduces single points of failure
• helps you scale
but…
• isolates applications from each other
• each QM is still a single point of failure
App App App App
App
Uniform Clusters

Moving from a single queue manager to multiple
queue managers:
• reduces single points of failure
• helps you scale
but…
• isolates applications from each other
• each QM is still a single point of failure
App App App App
App
App App App App
App
Uniform Clusters

Uniform Clusters App App App App
App
Improved by putting QMs into a cluster
• applications can find queues on different QMs
• adding new QMs becomes easier
but…
• affinities to specific QMs often still exist
• can end up with all applications connected to
a single QM
• nothing to stop individual QMs being
customised

Uniform Clusters App App App App
App
Cluster
App App App App
App
Improved by putting QMs into a cluster
• applications can find queues on different QMs
• adding new QMs becomes easier
but…
• affinities to specific QMs often still exist
• can end up with all applications connected to
a single QM
• nothing to stop individual QMs being
customised

Cluster
App App App App
App
Uniform clusters improve things further
• applications will spread themselves out evenly
• even if a queue manager goes down and
then comes back
• consistent configuration is easier to achieve
• affinities to specific QMs discouraged (and
can be prevented)
• enhances visibility of which applications you
have in your network
Uniform Clusters

Cluster
App App App App
App
Cluster
App App App App
App
decoupled
Uniform
Uniform clusters improve things further
• applications will spread themselves out evenly
• even if a queue manager goes down and
then comes back
• consistent configuration is easier to achieve
• affinities to specific QMs discouraged (and
can be prevented)
• enhances visibility of which applications you
have in your network
Uniform Clusters

Can I use this for all applications?
This pattern of decoupled applications works for certain application styles, ones that have little
affinity or have been designed appropriately for active/active deployments
Good
Applications that can handle being moved from one
queue manager to another without even realising
and can run with multiple instances
• Datagram producers, e.g. Events
• Services that respond to request messages
• No message ordering requirements
Bad
Applications that create persistent state across
multiple messaging operations, or require a single
instance to be running
• Requestors waiting for specific replies
• (Can be prevented in 9.2)
• Dependant on message ordering

Uniform Cluster Feature Set
Creation of a Uniform Cluster
• Putting QMs in a Uniform Cluster, and keeping their configuration consistent
Auto reconnectable applications
• Once in a Uniform Cluster, MQ can reconnect applications with a “hint” QM to reconnect to
• The hint will be based on which QMs can accommodate more applications
Identify applications by name, to define grouping of related applications for balancing
• Extends the existing JMS capability to all languages
Monitor application status across the entire cluster
• New APSTATUS command highlights how many instances are running, and where
• New system topic class STATAPP
Text based CCDTs to make it easier to maintain, and simpler to script cluster changes
• And to allows duplicate channel names
Prevent affinities to individual queue managers
• Helps maintain good practices

9.1.2, 9.1.4
9.1.2 – 9.1.4
9.1.2
9.1.3
9.1.2
9.2
9.1.5

9.1.2, 9.1.4
9.1.2 – 9.1.4
9.1.2
9.1.3
9.1.2
9.2
9.1.5
Let’s look at how things work
in more detail…

Before automatic application
balancing…
Many of you have built your own continuously
available and horizontally scalable solutions over the
years
This is the “uniform cluster” pattern
MQ has provided you many of the building blocks -
Client auto-reconnect
CCDT queue manager groups
MQ Clustering
But you had been left to solve some of the
problems, particularly with long running applications
-
Efficiently distributing your applications
Ensuring all messages are processed
Maintaining availability during maintenance
Handling growth and contraction of scale
App App App
decoupled
http://ibm.biz/MQ-UniCluster

Before automatic application
balancing…
Many of you have built your own continuously
available and horizontally scalable solutions over the
years
This is the “uniform cluster” pattern
MQ has provided you many of the building blocks -
Client auto-reconnect
CCDT queue manager groups
MQ Clustering
But you had been left to solve some of the
problems, particularly with long running applications
-
Efficiently distributing your applications
Ensuring all messages are processed
Maintaining availability during maintenance
Handling growth and contraction of scale
App App App
decoupled
App
App

Uniform Cluster
For the distributed platforms, declare a set of matching
queue managers to be following the uniform cluster pattern
All members of an MQ Cluster
Matching queues are defined on every queue manager
Applications can connect as clients to every queue manager
MQ will automatically share application connectivity
knowledge between queue managers
The group will use this knowledge to automatically keep
matching application instances balanced across the queue
managers
Matching applications are based on application name (new
abilities to programmatically define this)
MQ 9.1.2 started the roll out of Uniform Cluster support and
this has been continuing through 9.1.3 and 9.1.4, and on into
the future…
Uniform Clusters have started to make
that easier…
Application awareness

Uniform Cluster
Application awareness
Application instances can initially connect to any member of the group
We recommend you use a queue manager group and CCDT to
remove any SPoF
Every member of the uniform cluster will detect an imbalance and
request other queue managers to donate their applications
Hosting queue managers will instigate a client auto-reconnect with
instructions of where to reconnect to
Applications that have enabled auto-reconnect will automatically move
their connection to the indicated queue manager
(Client support has been increased over subsequent CD releases. 9.1.2
CD started with support for C-based applications, 9.1.3 CD added JMS
(Java SE) and 9.1.4 added .Net)
App App App App App App
Automatic Application Balancing

Uniform Cluster
Application instances can initially connect to any member of the group
We recommend you use a queue manager group and CCDT to
remove any SPoF
Every member of the uniform cluster will detect an imbalance and
request other queue managers to donate their applications
Hosting queue managers will instigate a client auto-reconnect with
instructions of where to reconnect to
Applications that have enabled auto-reconnect will automatically move
their connection to the indicated queue manager
(Client support has been increased over subsequent CD releases. 9.1.2
CD started with support for C-based applications, 9.1.3 CD added JMS
(Java SE) and 9.1.4 added .Net)

Uniform Cluster
Automatically handle rebalancing following
planned and unplanned queue manager outages
Existing client auto-reconnect and CCDT queue
manager groups will enable initial re-connection
on failure
Uniform Cluster rebalancing will enable
automatic rebalancing on recovery

Uniform Cluster
Even to horizontally scale out a queue
manager deployment
Simply add a new queue manager to
the uniform cluster
The new queue manager will detect an
imbalance of applications and request
its fair share

Setting up a Uniform
Cluster

Uniform Cluster Features in Detail
Identify applications by name, to define grouping of related applications for
balancing
Text based CCDTs to make it easier to maintain, and simpler to script cluster
changes
AutoCluster:
Type=Uniform
ClusterName=UNIDEMO
$ export MQAPPLNAME=
MY.SAMPLE.APP
{ "channel": [
{ "name": ”SVRCONN.CHANNEL",
Channels:
DefRecon=YES
$ export AMQ_BLOCK_RECONN_DYN_QUEUES=TRUE
DIS APSTATUS(MY.SAMPLE.APP)

Configuring a Uniform Cluster
To create a Uniform Cluster requires multiple things across all queue managers:
1. An MQ Cluster for inter-queue manager communication
2. Configuration to identify which cluster is a Uniform Cluster
3. Common queues defined across all members
4. The ability for applications to connect to any members using server connections (using a CCDT)
Uniform Cluster
ALTER QMGR REPOS(….
DEFINE CHANNEL(…..
DEFINE QLOCAL(….
….
DEFINE QLOCAL(….
….
DEFINE QLOCAL(….
….
configure configure configure
Uniform Cluster
AutoCluster:
Type=Uniform
ClusterName=UNIDEMO
DEFINE QLOCAL(….
….
deploy
Traditional approach
Configure each queue manager individually
New approach (MQ 9.1.4)
Configure once and deploy to all queue managers

MQ 9.1.4 introduced the following concepts
Automatic configuration of qm.ini at startup
Automatic configuration from an MQSC script at startup
Uniform Cluster configuration variables
Automatic Uniform Cluster creation
> crtmqm -p 1414 -ii /shared/uniclus.ini -ic /shared/uniclus.mqsc -iv
CONNAME=""127.0.0.1(1404)"” QMGRx
CONNAME=""127.0.0.1(1404)"” QMGRx
CONNAME=""127.0.0.1(1404)"” QMGRx
CONNAME=""127.0.0.1(1404)"” QMGRx
What’s in here and here + what this is set to

Automatic configuration of qm.ini at startup
CONNAME=""127.0.0.1(1404)"” QMGRx
• Can use to configure any ini parameters in qm.ini
• Enable exits
• Control channel behaviour
• Set TCP buffers
• etc…
• File is re-read every time QM starts
• Contents copied into real qm.ini
• qm.ini no longer updated manually – must update original file and restart QM

Automatic configuration from an MQSC script at startup
CONNAME=""127.0.0.1(1404)"” QMGRx
• Can use to run any MQSC script(s) at QM start time
• Define queues
• Set authority records
• Create topics
• etc…
• Can be a directory of scripts, run in alphabetical order
• File(s) are re-run against the QM every time it starts
• Can still run scripts & runmqsc commands at any time, but ideally only status and problem
diagnosis commands
• Ideally new objects and configuration changes made by editing .mqsc scripts and restarting

Uniform Cluster configuration variables
CONNAME=""127.0.0.1(1404)"” QMGRx
• Define name-value pairs that can be used in MQSC scripts run at startup.
• Use the name of the attribute in MQSC in place of a real value
• But – currently limited to the CONNAME and channel name fields of a cluster-receiver channel
e.g. DEFINE CHANNEL(UNICLUSTER.QM1) CHLTYPE(CLUSRCVR)
CLUSTER(MYCLUSTER) CONNAME(+CONNAME+)
• Name-value pairs are stored in qm.ini
• Also 2 new pre-defined name-value pairs – QMNAME and AUTOCL (see later slides)

Automatic Uniform Cluster creation
• New qm.ini stanza AutoCluster
• Every QM in the cluster has the same stanza contents
AutoCluster:
ClusterName=UNICLUS
Type=Uniform
Repository1Name=QMB
Repository1Conname=QMB.dnsname(1414)
Repository2Name=QMA
Repository2Conname=QMA.dnsname(1414)
QMA sees that its name
matches full-repo 2, so sets its
REPOS attribute to UNICLUS.
Also creates a cluster-sender to
repo 1.
QMB sees that its name
matches full-repo 1, so sets its
REPOS attribute to UNICLUS.
Also creates a cluster-sender to
repo 2.
QMC, QMD, QME etc… create a cluster-sender channel to
both of the repository queue managers using the conname
for each, with the cluster attribute set to UNICLUS

Bringing it all together
Auto Uniform Cluster stanza
defined in e.g.uniclus.ini
• Any other qm.ini configuration
changes added to uniclus.ini
• qm.ini no longer updated
manually
Common start-time config script, using name-value
pairs in place of real values.
• No need to define cluster-sender channels
(AutoCluster stanza does that automatically)
• Cluster receiver channel must be defined, but uses
insertion variables from -iv settings
• Object definitions, authority records etc. follow
cluster-receiver definition
AutoCluster:
ClusterName=UNICLUS
Type=Uniform
Repository1Conname=QMB.dnsname(1414)
Repository1Name=QMB
Repository2Conname=QMA.dnsname(1414)
Repository2Name=QMA
CONNAME=""127.0.0.1(1404)"” QMGRx

DEFINE CHANNEL(UNICLUSTER.+QMNAME+) CHLTYPE(CLUSRCVR) CLUSTER(+AUTOCL+) CONNAME(+CONNAME+)
DEFINE CHANNEL(SVRCONN.CHANNEL) CHLTYPE(SVRCONN)
DEFINE QLOCAL(Q1) CLUSTER(UNICLUSTER) DEFPSIST(YES) DEFBIND(NOTFIXED)
Now every QM in the cluster can be created using the same single command,
with CONNAME set to the address of this QM

Building scalable and available solutions
JSON CCDT
Build your own JSON format CCDTs
Supports multiple channels of the same name on
different queue managers to simplify the
building of uniform clusters
Available with all 9.1.2 clients
C, JMS, .NET, Node.js, Golang clients
01100110100101
10001010101101
10101011011011
01001011110111
01110111101111
01110111011
{
“channel”:[
{
“name”:”ABC”,
”queueManager”:”A”
},
{
“name”:”ABC”,
”queueManager”:”B”
},
]
}

Configuring the CCDT for application
balancing in a Uniform Cluster
To correctly setup a CCDT for application
rebalancing it needs to contain two entries per
queue manager:
• An entry under the name of a queue
manager group
• And entry under the queue manager’s
real name
(These previously would need to be different
channels, but with the JSON CCDT this is
unnecessary)
The application connects using the queue
manager group as the queue manager name
(prefixed with an ‘*’)
{
"channel":
[
{
"name": ”SVRCONN.CHANNEL",
"type": "clientConnection”
"clientConnection":
{
"connection":
[
{
"host": ”host1",
"port": 1414
}
],
"queueManager": "ANY_QM”
},
},
{
"clientConnection":
{
"connection":
[
{
"host": ”host2",
"port": 1414
}
],
"queueManager": "ANY_QM”
},
},
…
…
{
"clientConnection":
{
"connection":
[
{
"host": ”host1",
"port": 1414
}
],
"queueManager": ”QMGR1”
},
},
{
"clientConnection":
{
"connection":
[
{
"host": ”host2",
"port": 1414
}
],
"queueManager": ”QMGR2”
},
}
]
}
QMGR1
QMGR2

• Automatic application balancing is based on the
application name alone
• Different groups of application instances with
different application names are balanced
independently
• By default the application name is the executable
name
• This has been customisable with Java and JMS
applications for a while
• MQ 9.1.2 CD clients have extended this to other
programming languages
• For example C, .NET, XMS, …
• Application name can be set either programmatically
or as an environment override
Balancing by application name App App App App App App
App App App App
App App

58
View application status
• Now that MQ is taking a more application centric
view, a new command was added in MQ 9.1.3 to
Distributed mqsc to aid the understanding of
how applications are balanced across a Uniform
Cluster
• From any member of the Uniform Cluster, the
command displays applications by name and
highlights instances that are not evenly
balanced
DISPLAY APSTATUS(*) TYPE(APPL)
AMQ8932I: Display application status details.
APPLNAME(AMQSPHAC) CLUSTER(UNIDEMO)
COUNT(8) MOVCOUNT(8)
BALANCED(YES)
APPLNAME(AMQSPUTC) CLUSTER( )
BALANCED(NOTAPPLIC)
DISPLAY APSTATUS(*) TYPE(QMGR)
APPLNAME(AMQSPHAC) ACTIVE(YES)
BALSTATE(OK) LMSGDATE(2019-05-08)
LMSGTIME(14:05:36) QMNAME(UNID001)
QMID(UNID001_2019-05-08_13.59.31)
QMID(UNID002_2019-05-08_13.59.35)
QMID(UNID003_2019-05-08_13.59.40)
APPLNAME(AMQSPUTC) ACTIVE(YES)
BALSTATE(NOTAPPLIC) LMSGDATE(2019-05-08)
QMID(UNID001_2019-05-08_13.59.31)

Taking a QM out of the cluster
Sometimes you want to need to performance
maintenance on a queue manager. You can issue
‘endmqm –r’ and the cluster will rebalance
applications.
However, you might want to leave the QM running.
Before 9.2 this was complicated. With 9.2 things
are much simpler:
SUSPEND QMGR CLUSTER(MYCLUSTER)
RESUME QMGR CLUSTER(MYCLUSTER)
59

applications.
are much simpler:
60

applications.
are much simpler:
61

applications.
are much simpler:
62

applications.
are much simpler:
63

applications.
are much simpler:
64

QM1 QM2 QM3
Do I still need an availability solution? App App App App
App
decoupled
Uniform Cluster
QM1 QM2 QM3
• Remember that a Uniform Cluster
is just an MQ Cluster. It improves
your service availability.
• The queue managers in the cluster
are distinct from each other.
Messages on QM1 don’t exist on
QM2 or QM3.
• If you need high message
availability, you still need to ensure
each QM can be restored quickly

Uniform Clusters
RDQM HA Group
Node 2 Node 3
Node 1
QM 1
App
Network
App App
Uniform
Cluster
• Remember that a Uniform Cluster
is just an MQ Cluster. It improves
your service availability.
• The queue managers in the cluster
are distinct from each other.
Messages on QM1 don’t exist on
QM2 or QM3.
• If you need high message
availability, you still need to ensure
each QM can be restored quickly

Thank you
Matthew Whitehead
MQ Architect, Distributed Platforms
—
mwhitehead@uk.ibm.com
© Copyright IBM Corporation 2019. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express
or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and ibm.com are trademarks of IBM
Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarksof IBM or other companies. A current list of IBM trademarksis available at Copyright and trademark
information.

Building an Active-Active IBM MQ System

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Building an Active-Active IBM MQ System

Similaire à Building an Active-Active IBM MQ System (20)

Plus de matthew1001

Plus de matthew1001 (13)

Dernier

Dernier (20)

Building an Active-Active IBM MQ System