SlideShare a Scribd company logo
1 of 42
Download to read offline
HHM-2833: Where is My Message?: Using
IBM MQ Tools to Work Out What Applications
Have Done
Matt Leming – lemingma@uk.ibm.com
IBM Hursley - UK
Symptoms include:
• Timeouts for request/response messaging
• Inconsistent data between front-end and back-end systems
• Customer complaints about ‘missing’ transactions
1
Agenda – the MQ Toolbox
• Who’s connected?
• Are messages flowing?
• Where are messages going?
• What are the apps doing?
• How can I look back in time? Presentation contains MQSC
example commands and output.
Your own admin tools will
vary by task, situation and
preference. Tasks can be
performed in PowerShell, MQ
Explorer etc. as desired.
2
Who’s connected?
DISPLAY CONN
DISPLAY CONN(*) TYPE(HANDLE) ALL DIS CONN(*) ALL
AMQ8276: Display Connection details.
CONN(577C425321295301)
EXTCONN(414D5143474154455741593120202020)
TYPE(HANDLE)
OBJNAME(WLMMDB.REQUEST) OBJTYPE(QUEUE)
ASTATE(NONE) HSTATE(INACTIVE)
OPENOPTS(MQOO_OUTPUT,MQOO_FAIL_IF_QUIESCING)
READA(NO)
OBJNAME(SENDINGAPP.REPLY) OBJTYPE(QUEUE)
ASTATE(ACTIVE) HSTATE(ACTIVE)
OPENOPTS(MQOO_INPUT_SHARED,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING)
READA(NO)
AMQ8276: Display Connection details.
CONN(577C425321295301)
EXTCONN(414D5143474154455741593120202020)
TYPE(CONN)
PID(9740) TID(185)
APPLDESC(WebSphere MQ Channel) APPLTAG(jms/GATEWAY1_CF)
APPLTYPE(SYSTEM) ASTATE(NONE)
CHANNEL(WAS.CLIENTS) CONNAME(127.0.0.1)
CONNOPTS(MQCNO_SHARED_BINDING) USERID(pbroad)
UOWLOG( ) UOWSTDA(2014-04-08)
UOWSTTI(13.24.00) UOWLOGDA( )
UOWLOGTI( ) URTYPE(XA)
EXTURID(XA_FORMATID[DSAW]
XA_GTRID[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809]
XA_BQUAL[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809000000010000000000000000
000000000001])
QMURID(0.7940075) UOWSTATE(ACTIVE)
Use CONN to match TYPE(CONN) and
TYPE(HANDLE) records
TYPE(HANDLE) records let you find
applications by the objects they access.
See all open handles for an app in one place,
unlike DIS QSTATUS records
Long running UOW information.
XID can be tied up with app server txn timeout
Channel name + IP help identify client apps.
MQ V7.5 and later JMS clients can supply
an application name in the CF
4
DISPLAY CHSTATUS
DISPLAY CHSTATUS(*) ALL
AMQ8417: Display Channel Status details.
CHANNEL(WAS.CLIENTS) CHLTYPE(SVRCONN)
BUFSRCVD(17) BUFSSENT(13)
BYTSRCVD(2296) BYTSSENT(2456)
CHSTADA(2014-04-08) CHSTATI(15.26.59)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(127.0.0.1) CURRENT
EXITTIME(0,0) HBINT(5)
JOBNAME(0000260C000000B9) LOCLADDR( )
LSTMSGDA(2014-04-08) LSTMSGTI(15.26.59)
MCASTAT(RUNNING) MCAUSER(pbroad)
MONCHL(OFF) MSGS(6)
RAPPLTAG(jar) SSLCERTI(CN=ExampleCA,O=Example)
SSLKEYDA( ) SSLKEYTI( )
SSLPEER(SERIALNUMBER=53:43:FD:D6,CN=ExampleApp1,O=Example)
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(RECEIVE)
CURSHCNV(1) MAXSHCNV(1)
RVERSION(00000000) RPRODUCT(MQJM)
See SSLPEER information not in DIS CONN
JOBNAME contains PID (except z/OS): 0x260C = PID(9740)
On Linux/UNIX (not Win) TID matches CONN: 0xB9 = TID(185)
Note that multiple CONN might share one SVRCONN channel instance
Check suitable heartbeats are negotiated
5
Automatic detection if no-one is
connected
AMQSCLM
1) Normal operation
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
Round-robin message distribution
App
server 1
App
server 2
App
server N
. . .
IPPROCS(>=1) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM AMQSCLM AMQSCLM
7
AMQSCLM
2) Application failure
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
Messages
start
building
up
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
8
AMQSCLM
3) Failure Detection
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM
regularly
checks the
queue and
detects
IPPROCS(0)
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
9
AMQSCLM
4) Redirect Messages
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM
reduces
queue priority
in cluster, to
stop new
messages
arriving
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
10
AMQSCLM
5) Un-maroon Msgs (optional)
Driver
qmgr 1
Driver
qmgr 2
Driver
qmgr N
. . .
Target
qmgr 1
Target
qmgr 2
Target
qmgr n
. . .
Sending appsSending appsSending apps
Cluster
WLM
App
server 1
App
server 2
. . .
IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1)
AMQSCLM
tells qmgr to
redistribute
‘stuck’
messages via
the cluster
AMQSCLM AMQSCLM AMQSCLM
App
server N
Round-robin message distribution
11
AMQSCLM summary
• The Cluster Queue Monitoring sample program (AMQSCLM)
• Shipped with the product as a sample
– Precompiled
– Source code
– Platforms except on z/OS
• More information here:
http://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.
mq.dev.doc/q024620_.htm
12
Are messages flowing?
Real-time/online Monitoring – Queues
• Set detail level for QMGR. Override for individual Queues
• Gives live view of application responsiveness
ALTER QMGR MONQ(MEDIUM)
ALTER QLOCAL(QUEUE1) MONQ(HIGH)
ALTER QLOCAL(QUEUE2) MONQ(OFF)
AMQ8450: Display queue status details.
QUEUE(QUEUE1) TYPE(QUEUE)
CURDEPTH(16) IPPROCS(3)
LGETDATE(2014-04-08) LGETTIME(17.05.59)
LPUTDATE(2014-04-08) LPUTTIME(17.12.16)
MEDIALOG( ) MONQ(HIGH)
MSGAGE(112) OPPROCS(5)
QTIME(10101414, 10101414) UNCOM(NO)
DIS QSTATUS(QUEUE1) ALL
Age in seconds of the oldest
message on the queue
Estimations of the time in microseconds that messages are waiting on the queue for
processing.
First value: Calculated from recent activity
Second value: Calculated from longer term activity
Timestamps of last PUT/GET
to check for recent activity
Without MONQ you only get the depth and
how many handles are open
14
Real-time/online Monitoring – Channels
• Gives live view of channel throughput
ALTER QMGR MONCHL(MEDIUM)
ALTER CHANNEL(CLUSTER1.QM1) CHLTYPE(CLUSRCVR) MONCHL(HIGH)
AMQ8417: Display Channel Status details.
CHANNEL(MQHUB.GATEWAY2) CHLTYPE(CLUSSDR)
BATCHES(52) BATCHSZ(50)
BUFSRCVD(55) BUFSSENT(1616)
BYTSRCVD(1748) BYTSSENT(1192330)
CHSTADA(2014-04-08) CHSTATI(17.49.03)
COMPHDR(NONE,NONE) COMPMSG(NONE,NONE)
COMPRATE(0,0) COMPTIME(0,0)
CONNAME(127.0.0.1(1422)) CURLUWID(0107445310001B34)
CURMSGS(50) CURRENT
CURSEQNO(11823) EXITTIME(0,0)
HBINT(5) INDOUBT(YES)
JOBNAME(00002DA4000047D0) LOCLADDR(127.0.0.1(53557))
LONGRTS(999999999) LSTLUWID(0107445310001B33)
LSTMSGDA(2014-04-08) LSTMSGTI(17.49.51)
LSTSEQNO(11773) MCASTAT(RUNNING)
MONCHL(MEDIUM) MSGS(1580)
NETTIME(137538,29555) NPMSPEED(FAST)
RQMNAME(GATEWAY2) SHORTRTS(180)
SSLCERTI( ) SSLKEYDA( )
SSLKEYTI( ) SSLPEER( )
SSLRKEYS(0) STATUS(RUNNING)
STOPREQ(NO) SUBSTATE(RECEIVE)
XBATCHSZ(20,17) XMITQ(SYSTEM.CLUSTER.TRANSMIT.QUEUE)
XQMSGSA(112) XQTIME(545784,3929968)
RVERSION(07050002) RPRODUCT(MQMM)
DIS CHSTATUS(MQHUB.GATEWAY2) ALL
Short/long term calculations of how
full your batches are getting, to help
you tune BATCHSZ/BATCHINT
Last time a message was sent over
the channel
Depth of messages on XMITQ
for this channel (capped at 999)
Short/long term calculations of how
long messages are waiting on the
XMITQ for transmission
15
Accounting and Statistics Overview
• Monitoring data sent as a PCF message at a configured interval
– Statistics – scoped to a Queue / Channel / QMGR
– Accounting – scoped to an individual CONN and Queue / QMGR
• Applications must consume the messages
– Enterprise monitoring – Tivoli ITCAM / Tivoli OMEGAMON XE for Messaging
– Sample applications and SupportPacs
– Custom applications – com.ibm.mq.headers Java package
PCF accounting
collection app
Queues Channels
CONNs
PCF statistics
collection app
Statistics
Messages
Accounting
Messages
QMGR
SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE
16
Taking a look with SupportPac MS0P
• Time period
• MQ Statistics at QMGR level
• Detailed queue statistics
17
Simple practical example using amqsmon
• Overall QMGR busyness
• Simple data format
– Multiple values are
[Persistent, NonPersistent]
• One message every X seconds
– Use amqsmon directly (perl/cron)
• Low/high water marks for
subscriptions
– Grouped by subscription type
ALTER QMGR STATMQI(ON)
Wait a bit
RESET QMGR TYPE(STATISTICS)
amqsmon -m GATEWAY1 -t statistics -a -w 0
MonitoringType: MQIStatistics
QueueManager: 'GATEWAY1'
IntervalStartDate: '2014-04-09'
IntervalStartTime: '00.00.35'
IntervalEndDate: '2014-04-09'
IntervalEndTime: '00.01.13'
CommandLevel: 700
ConnCount: 35
PutCount: [271, 0]
PutFailCount: 0
Put1Count: [2, 0]
Put1FailCount: 0
PutBytes: [273976, 0]
GetCount: [270, 0]
GetBytes: [269468, 0]
GetFailCount: 19
DurableSubscriptionHighWater: [0, 0, 0, 0]
DurableSubscriptionLowWater: [0, 0, 0, 0]
NonDurableSubscriptionHighWater: [0, 0, 0, 0]
NonDurableSubscriptionLowWater: [0, 0, 0, 0]
PutTopicCount: [0, 0]
PutTopicFailCount: 0
Put1TopicCount: [0, 0]
Put1TopicFailCount: 0
PutTopicBytes: [0, 0]
PublishMsgCount: [0, 0]
PublishMsgBytes: [0, 0]
18
Where are messages going?
Cluster
MQ networks can be complex
• At each of the dots stuck / mis-sent messages are possible
– MQOPENs of the wrong Queue / Queue Manager by apps
– Full queues
– Stopped channels
– Stopped apps
– Incorrectly configured QREMOTE/QALIAS routing objects
– Cluster membership problems
• The standard problem diagnosis approach
– Methodically checking channels/queues/DLQs at each point
• Is there anything to speed up this process?
App1
App2A
App2B
Requests Replies
Replies
SDR/RCVR
SDR/RCVR
20
Cluster
Trace Route
• MQ has the ability to inject tracer messages
– (Can be) hidden from applications
– Generate activity reports as they pass through
• Tools are available to trace routes using these reports
– dspmqrte – command line tool supplied with the product
– MS0P – Cat 2 SupportPac extension to MQ Explorer
– Report messages need to get back to your ReplyQ of your tool
• Lets you see the path messages could have taken
– Test connectivity through the MQ network
– Test cluster workload balancing
• Can quickly jump you close to the problem
– The point your trace message veers off in the wrong direction
– The point the trail goes cold
dspmqrte SDR/RCVR
SDR/RCVR
dspmqrte
dspmqrte
21
What are the apps doing?
Application Activity Tracing
• Similar infrastructure to Accounting & Statistics
– PCF messages on SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE
– Configurable via mqat.ini
• Can be changed without QMGR restart
• Configurable detail level can include partial/full message payload
• Frequency options for tuning
– Can be enabled on a per-application basis
• Via MQCONNX flags
• Via application name
– In MQ V7.5 and later JMS clients can supply an application
name in the connection factory
Health warning: Performance impact
http://ow.ly/vA8wB
23
Application Activity Tracing
• Enables scenarios such as
– Application audit trail
– Message duplication
– Resource usage
• Which queues or topics are actually being used
– Problem Determination
• Which Queue / Queue Manager is the application
actually opening
– Application Coding Standards
• Does everyone use the MQI in the recommended way
– And more …
Health warning: Performance impact
http://ow.ly/vA8wB
24
Looking at the data with the amqsact sample
ALTER QMGR ACTVTRC(ON) <- should be tuned via mqat.ini
amqsact -m GATEWAY1 -v
MQI Operation: 6
Operation Id: MQXF_PUT
ApplicationTid: 12451
OperationDate: '2014-04-09'
OperationTime: '01:39:48'
High Res Time: 1397003988665548
Completion Code: MQCC_OK
Reason Code: 0
Hobj: 18225032
Put Options: 139330
Msg length: 460
Recs_present: 0
Known_dest_count: 1
Unknown_dest_count: 0
Invalid_dest_count: 0
Object_type: MQOT_Q
Object_name: 'SENDINGAPP.REPLY'
Object_Q_mgr_name: 'GATEWAY1'
Resolved_Q_Name: 'SENDINGAPP.REPLY'
Resolved_Q_mgr: 'GATEWAY1'
Resolved_local_Q_name: 'SENDINGAPP.REPLY'
Resolved_local_Q_mgr: 'GATEWAY1'
Resolved_type: MQOT_Q
Report Options: 0
Msg_type: MQMT_DATAGRAM
Expiry: -1
Format_name: 'MQHRF2'
Priority: 4
Persistence: 0
Msg_id:
00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 '
00000010: 0207 4453 2007 2603 '..DS .&. '
Correl_id:
00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 '
00000010: 0207 4453 2007 2203 '..DS .". '
Reply_to_Q : ' ^D'
Reply_to_Q_Mgr: ' ^C'
Coded_char_set_id: 1208
Encoding: 273
Put_date: '20140409'
Put_time: '00394866'
Check the options used for coding standards
Check queue name resolution, to find out
why messages are going to the wrong place
Track individual messages and request/reply
scenarios with Msg_id and Correl_id
25
Extract from Report
MonitoringType: MQI Activity Trace
QueueManager: 'V71'
Host Name: 'rockall.hursley.ibm.com'
CommandLevel: 710
ApplicationName: 'WebSphere MQ Client for Java'
ApplicationPid: 18612354
UserId: 'mquser'
ConnName: '9.20.95.106'
Channel Type: MQCHT_SVRCONN
Platform: MQPL_UNIX
=======================================================================
=
Time Operation CompCode MQRC HObj (ObjName)
10:04:09 MQXF_INQ MQCC_OK 0000 2
10:04:09 MQXF_CLOSE MQCC_OK 0000 2
10:04:09 MQXF_OPEN MQCC_OK 0000 4 ()
10:04:09 MQXF_INQ MQCC_OK 0000 4
10:04:09 MQXF_CLOSE MQCC_OK 0000 4
10:04:09 MQXF_OPEN MQCC_OK 0000 4 (SYSTEM.DEFAULT.LOCAL.QUEUE)
10:04:09 MQXF_INQ MQCC_OK 0000 4
26
How can I look back in time?
What happened to my messages at 2am this
morning?
• Enterprise monitoring solution
– DLQ alerts, queue depth alerts, channel status alerts
– Unresolved running units of work
– Historical MQ monitoring, accounting and stats data
• App logs from the time of the problem
– Exceptions, MQ error codes, timeouts
• MQ error logs for all qmgrs that could have been involved
– Channel errors
– Authentication issues
• ??? – what else is there
29
What about the MQ Recovery Log?
• For persistent messages inside transactions
– MQ logs each operation performed
• Why can't we use this to
– Look back in time to 2am and see what happened?
– Recovery the original payload if the app lost the message?
– See what happened inside long-running units of work?
– Provide a list of operations within the failed business transaction?
• MQ documents how you can… if
– You use the text formatting tool provided with MQ (dmpmqlog)
– The logging is linear so the historical data is available in the tool
– You follow the right steps to extract data from running qmgrs
– You do the work to follow through the logs
30
Dmpmqlog output is readable, but analysis is tedious
• Wouldn't it be easier to let
the computer do the
tedious bit?
LOG RECORD - LSN <0:0:954:44817>
**********
HLG Header: lrecsize 873, version 1, rmid 0, eyecatcher HLRH
LogRecdType . . : AQM Put Message (257)
Eyecatcher . . : ALRH Version . . . . : 1
LogRecdLen . . : 853 LogRecdOwnr . . : 256 (AQM)
XTranid . . . . : TranType: XA
XID: formatID 1463898948, gtrid_length 36, bqual_length 54
gtrid [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F2]
bqual
[000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F20000000100000000000
00000000000000001]
QueueName . . . : Not known
Qid . . . . . . : {Hash 2147211283, Counter: 5}
ThisLSN . . . . : <0:0:0:0>
PrevLSN . . . . : <0:0:954:43944>
Version . . . . : 4
MapIndex . . . : 199
PrevLink.Locn . : 102408 PrevLink.Length : 8
PrevDataLink . : {High 0, Low 103424}
Data.Locn . . . : 103424 Data.Length . . : 613
Data . . . . . :
00000: 41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF AQRH....ÿÿÿÿÿÿÿÿ
00016: 00 00 00 00 00 00 00 00 C7 00 00 00 02 00 C0 01 ........Ç.......
00032: 00 00 00 00 04 00 01 00 A5 00 00 00 00 00 00 00 ........¥.......
00048: 63 00 00 00 41 4D 51 20 49 49 42 30 31 5F 51 4D c...AMQ IIB01_QM
00064: 20 20 20 20 D9 26 B0 52 20 08 53 F5 30 30 30 30 .&.R .Sõ0000
00080: 30 30 39 39 00 00 00 00 00 00 00 00 00 00 00 00 0099............
00096: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00112: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................
00128: 00 00 00 00 FF FF FF FF 00 00 00 00 04 00 00 09 ....ÿÿÿÿ........
00144: 00 00 00 00 E3 ED 04 80 10 FD 67 E4 FF FF FF FF ....ãí.€.ýgäÿÿÿÿ
00160: 4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00 MD ............
00176: 00 00 00 00 11 01 00 00 B8 04 00 00 4D 51 48 52 ........¸...MQHR
00192: 46 32 20 20 04 00 00 00 01 00 00 00 20 20 20 20 F2 ........
00208: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00224: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00240: 20 20 20 20 20 20 20 20 20 20 20 20 49 49 42 30 IIB0
00256: 31 5F 51 4D 20 20 20 20 20 20 20 20 20 20 20 20 1_QM
00272: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00288: 20 20 20 20 20 20 20 20 20 20 20 20 4D 55 53 52 MUSR
00304: 5F 4D 51 41 44 4D 49 4E 16 01 05 15 00 00 00 64 _MQADMIN.......d
00320: 20 3E AC 57 48 B3 09 B8 71 B0 4C F2 03 00 00 00 >¬WH..¸q.Lò....
00336: 00 00 00 00 00 00 00 0B 20 20 20 20 20 20 20 20 ........
00352: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
00368: 20 20 20 20 20 20 20 20 1C 00 00 00 57 65 62 53 ....WebS
00384: 70 68 65 72 65 20 4D 51 20 43 6C 69 65 6E 74 20 phere MQ Client
00400: 66 6F 72 20 4A 61 76 61 32 30 31 33 31 32 31 39 for Java20131219
00416: 31 34 32 32 33 32 32 32 20 20 20 20 00 00 00 00 14223222 ....
00432: A4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ¤...............
00448: 52 46 48 20 00 00 00 02 00 00 00 A4 00 00 01 11 RFH .......¤....
00464: 00 00 04 B8 4D 51 53 54 52 20 20 20 00 00 00 00 ...¸MQSTR ....
00480: 00 00 04 B8 00 00 00 20 3C 6D 63 64 3E 3C 4D 73 ...¸... <mcd><Ms
00496: 64 3E 6A 6D 73 5F 74 65 78 74 3C 2F 4D 73 64 3E d>jms_text</Msd>
00512: 3C 2F 6D 63 64 3E 20 20 00 00 00 58 3C 6A 6D 73 </mcd> ...X<jms
00528: 3E 3C 44 73 74 3E 71 75 65 75 65 3A 2F 2F 2F 51 ><Dst>queue:///Q
00544: 31 3C 2F 44 73 74 3E 3C 54 6D 73 3E 31 33 38 37 1</Dst><Tms>1387
00560: 34 36 32 39 35 32 32 32 36 3C 2F 54 6D 73 3E 3C 462952226</Tms><
00576: 43 69 64 3E 30 30 30 30 30 30 39 39 3C 2F 43 69 Cid>00000099</Ci
00592: 64 3E 3C 44 6C 76 3E 32 3C 2F 44 6C 76 3E 3C 2F d><Dlv>2</Dlv></
Ordered unique IDs for each record (LSN)
A set of documented record types
Transaction information with XIDs, or re-used
MQ transaction IDs
MQMD header data at discoverable offsets
in the hex of a message Put
The message payload itself
31
Check out dmpmqlog scraper tool
• Takes the tedium out of analysing the output from dmpmqlog
• Created by Peter Broadhurst
• Download from
http://www.ibm.com/support/docview.wss?uid=swg21660642
• Generates file per message PUT in the supplied data.
• Summary file
java -jar dmpmqlog.scraper-20151201.jar -b little-endian -i dmpmqlog.txt -o .
32
dmpmqlog scraper tool output
• Generates file per message PUT in the supplied data.
• Summary file
33
Summary
• Lots of tools in your MQ toolbox!
• On-line Status Commands
– DISPLAY CONN
– DISPLAY QSTATUS
– DISPLAY CHSTATUS
• Cluster monitoring – AMQSCLM
• Off-line Statistics and Accounting
– amqsmon and MS0P to view
• Tracking
– Trace-route
– Application Activity Trace
• MQ Recovery Logs
– dmpmqlog scraper
34
Where can I get more information?
IBM Messaging developerWorks
developer.ibm.com/messaging
IBM Messaging Youtube
https://www.youtube.com/IBMmessagingMedia
LinkedIn
Ibm.biz/ibmmessaging
Twitter
@IBMMessaging
IBM MQ Facebook
Facebook.com/IBM-MQ-8304628654/
35
Monday
10:30-11:30 3592 New MQ features
3452 Managing applications
12:00-13:00 2835 MQ on z/OS and Distributed
15:00-16:00 3470 Latest MQ z/OS features
2833 Where is my message?
3544 MQ Light in an MQ infrastructure
16:30-17:30 3573 Hybrid cloud messaging
2941 MQ Advanced
Tuesday
08:30-09:30 3540 The MQ Light API
12:00-13:00 3456 The IBM MQ Appliance
13:15-14:15 3499 Introducing Message Hub
3458 MQ Appliance administration
14:30-15:30 6432 MQ updates and futures (InnerCircle)
2849 Messaging feedback roundtable
16:00-17:00 3544 MQ Light in an MQ infrastructure
3513 MQ hands on lab
Wednesday
08:30-09:30 3602 Managing your MQ environment
12:00-13:00 3613 Designing MQ self service
6408 Hybrid messaging roadmap (InnerCircle)
13:15-14:00 3416 HA and DR with MQ
3433 Why secure your messaging?
15:45-16:30 3429 Securing MQ
2847 Meet the messaging experts
16:00-17:00 3508 MQ Light hands on lab
16:45-17:30 2275 Migrating to the IBM MQ Appliance
Thursday
08:30-09:15 3420 MQ Clustering
2931 Business agility with self service MQ
09:30-10:15 3479 MQ z/OS clusters and shared queue
3450 Optimising MQ applications
2849 Messaging feedback roundtable
10:30-11:15 3465 MQ Appliance high availability
3481 MQ z/OS messaging connectivity
11:30-12:15 3474 Active-active messaging
3537 Monitoring and managing MQ
3425 MQ publish/subscribe
Find us at the EXPO:
Hybrid Integration peds 65-68
Check out the Hybrid Messaging sub topic under the
Hybrid Integration topic for further customer and business
partner sessions
Hybrid Messaging from the IBM experts at InterConnect 2016
Sunday
14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)
MQ as a service Redpaper now available
• IBM MQ as a Service: A Practical Approach
• Provides information about how to build, deploy, and use IBM MQ as a
service. The information in this paper includes the key factors that must
be considered while planning the use of IBM MQ as a service
• http://ibm.biz/mqaas_red
Questions?
Notices and Disclaimers
40
Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission
from IBM.
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of
initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS
DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE
USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY.
IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers
have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in
which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and
discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their
specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and
interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such
laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
Notices and Disclaimers Con’t.
41
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.

More Related Content

What's hot

IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction ejlp12
 
IBM MQ Whats new - including 9.3 and 9.3.1
IBM MQ Whats new - including 9.3 and 9.3.1IBM MQ Whats new - including 9.3 and 9.3.1
IBM MQ Whats new - including 9.3 and 9.3.1Robert Parker
 
IBM Websphere MQ Basic
IBM Websphere MQ BasicIBM Websphere MQ Basic
IBM Websphere MQ BasicPRASAD BHATKAR
 
IBM MQ Update, including 9.1.2 CD
IBM MQ Update, including 9.1.2 CDIBM MQ Update, including 9.1.2 CD
IBM MQ Update, including 9.1.2 CDDavid Ware
 
Deploying and managing IBM MQ in the Cloud
Deploying and managing IBM MQ in the CloudDeploying and managing IBM MQ in the Cloud
Deploying and managing IBM MQ in the CloudRobert Parker
 
Handle Large Messages In Apache Kafka
Handle Large Messages In Apache KafkaHandle Large Messages In Apache Kafka
Handle Large Messages In Apache KafkaJiangjie Qin
 
NATS Streaming - an alternative to Apache Kafka?
NATS Streaming - an alternative to Apache Kafka?NATS Streaming - an alternative to Apache Kafka?
NATS Streaming - an alternative to Apache Kafka?Anton Zadorozhniy
 
IBM MQ High Availability 2019
IBM MQ High Availability 2019IBM MQ High Availability 2019
IBM MQ High Availability 2019David Ware
 
Event Driven Systems with Spring Boot, Spring Cloud Streams and Kafka
Event Driven Systems with Spring Boot, Spring Cloud Streams and KafkaEvent Driven Systems with Spring Boot, Spring Cloud Streams and Kafka
Event Driven Systems with Spring Boot, Spring Cloud Streams and KafkaVMware Tanzu
 
The RabbitMQ Message Broker
The RabbitMQ Message BrokerThe RabbitMQ Message Broker
The RabbitMQ Message BrokerMartin Toshev
 
REST API Best (Recommended) Practices
REST API Best (Recommended) PracticesREST API Best (Recommended) Practices
REST API Best (Recommended) PracticesRasheed Waraich
 
Building APIs with Amazon API Gateway
Building APIs with Amazon API GatewayBuilding APIs with Amazon API Gateway
Building APIs with Amazon API GatewayAmazon Web Services
 
멀티 클라우드 시대의 정보보호 관리체계
멀티 클라우드 시대의 정보보호 관리체계멀티 클라우드 시대의 정보보호 관리체계
멀티 클라우드 시대의 정보보호 관리체계Logpresso
 
Building an Active-Active IBM MQ System
Building an Active-Active IBM MQ SystemBuilding an Active-Active IBM MQ System
Building an Active-Active IBM MQ Systemmatthew1001
 
IBM MQ - What's new in 9.2
IBM MQ - What's new in 9.2IBM MQ - What's new in 9.2
IBM MQ - What's new in 9.2David Ware
 
Build and Manage Your APIs with Amazon API Gateway
Build and Manage Your APIs with Amazon API GatewayBuild and Manage Your APIs with Amazon API Gateway
Build and Manage Your APIs with Amazon API GatewayAmazon Web Services
 

What's hot (20)

IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction IBM WebSphere MQ Introduction
IBM WebSphere MQ Introduction
 
IBM MQ Whats new - including 9.3 and 9.3.1
IBM MQ Whats new - including 9.3 and 9.3.1IBM MQ Whats new - including 9.3 and 9.3.1
IBM MQ Whats new - including 9.3 and 9.3.1
 
IBM Websphere MQ Basic
IBM Websphere MQ BasicIBM Websphere MQ Basic
IBM Websphere MQ Basic
 
Amazon simple queue service
Amazon simple queue serviceAmazon simple queue service
Amazon simple queue service
 
IBM MQ Update, including 9.1.2 CD
IBM MQ Update, including 9.1.2 CDIBM MQ Update, including 9.1.2 CD
IBM MQ Update, including 9.1.2 CD
 
Deploying and managing IBM MQ in the Cloud
Deploying and managing IBM MQ in the CloudDeploying and managing IBM MQ in the Cloud
Deploying and managing IBM MQ in the Cloud
 
Handle Large Messages In Apache Kafka
Handle Large Messages In Apache KafkaHandle Large Messages In Apache Kafka
Handle Large Messages In Apache Kafka
 
NATS Streaming - an alternative to Apache Kafka?
NATS Streaming - an alternative to Apache Kafka?NATS Streaming - an alternative to Apache Kafka?
NATS Streaming - an alternative to Apache Kafka?
 
IBM MQ High Availability 2019
IBM MQ High Availability 2019IBM MQ High Availability 2019
IBM MQ High Availability 2019
 
Event Driven Systems with Spring Boot, Spring Cloud Streams and Kafka
Event Driven Systems with Spring Boot, Spring Cloud Streams and KafkaEvent Driven Systems with Spring Boot, Spring Cloud Streams and Kafka
Event Driven Systems with Spring Boot, Spring Cloud Streams and Kafka
 
Deep Dive on AWS Lambda
Deep Dive on AWS LambdaDeep Dive on AWS Lambda
Deep Dive on AWS Lambda
 
AWS SQS SNS
AWS SQS SNSAWS SQS SNS
AWS SQS SNS
 
The RabbitMQ Message Broker
The RabbitMQ Message BrokerThe RabbitMQ Message Broker
The RabbitMQ Message Broker
 
REST API Best (Recommended) Practices
REST API Best (Recommended) PracticesREST API Best (Recommended) Practices
REST API Best (Recommended) Practices
 
Architecting for AWS
Architecting for AWSArchitecting for AWS
Architecting for AWS
 
Building APIs with Amazon API Gateway
Building APIs with Amazon API GatewayBuilding APIs with Amazon API Gateway
Building APIs with Amazon API Gateway
 
멀티 클라우드 시대의 정보보호 관리체계
멀티 클라우드 시대의 정보보호 관리체계멀티 클라우드 시대의 정보보호 관리체계
멀티 클라우드 시대의 정보보호 관리체계
 
Building an Active-Active IBM MQ System
Building an Active-Active IBM MQ SystemBuilding an Active-Active IBM MQ System
Building an Active-Active IBM MQ System
 
IBM MQ - What's new in 9.2
IBM MQ - What's new in 9.2IBM MQ - What's new in 9.2
IBM MQ - What's new in 9.2
 
Build and Manage Your APIs with Amazon API Gateway
Build and Manage Your APIs with Amazon API GatewayBuild and Manage Your APIs with Amazon API Gateway
Build and Manage Your APIs with Amazon API Gateway
 

Similar to HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applications Have Done

Where is my MQ message on z/OS?
Where is my MQ message on z/OS?Where is my MQ message on z/OS?
Where is my MQ message on z/OS?Matt Leming
 
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...Peter Broadhurst
 
Monitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systemsMonitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systemsmatthew1001
 
Where is My Message?: Use MQ Tools to Work Out What Applications Have Done
Where is My Message?: Use MQ Tools to Work Out What Applications Have DoneWhere is My Message?: Use MQ Tools to Work Out What Applications Have Done
Where is My Message?: Use MQ Tools to Work Out What Applications Have DoneMorag Hughson
 
Queue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docxQueue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docxsarvank2
 
Websphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsWebsphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsBiju Nair
 
Using Transaction Tracing to Determine Issues with Remote MQ Transactions
Using Transaction Tracing to Determine Issues with Remote MQ TransactionsUsing Transaction Tracing to Determine Issues with Remote MQ Transactions
Using Transaction Tracing to Determine Issues with Remote MQ TransactionsjKool
 
MSMQ - Microsoft Message Queueing
MSMQ - Microsoft Message QueueingMSMQ - Microsoft Message Queueing
MSMQ - Microsoft Message QueueingPeter R. Egli
 
5th KuVS Meeting
5th KuVS Meeting5th KuVS Meeting
5th KuVS Meetingsteccami
 
Network-Connected Development with ZeroMQ
Network-Connected Development with ZeroMQNetwork-Connected Development with ZeroMQ
Network-Connected Development with ZeroMQICS
 
Functional and non-functional testing with IoT-Testware
Functional and non-functional testing with IoT-TestwareFunctional and non-functional testing with IoT-Testware
Functional and non-functional testing with IoT-TestwareAxel Rennoch
 
Overview of ZeroMQ
Overview of ZeroMQOverview of ZeroMQ
Overview of ZeroMQpieterh
 
Timing attacks against web applications: Are they still practical?
Timing attacks against web applications: Are they still practical?Timing attacks against web applications: Are they still practical?
Timing attacks against web applications: Are they still practical?DefCamp
 
RTOS implementation
RTOS implementationRTOS implementation
RTOS implementationRajan Kumar
 
3450 - Writing and optimising applications for performance in a hybrid messag...
3450 - Writing and optimising applications for performance in a hybrid messag...3450 - Writing and optimising applications for performance in a hybrid messag...
3450 - Writing and optimising applications for performance in a hybrid messag...Timothy McCormick
 
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)Jakub Botwicz
 
Webinar Monitoring in era of cloud computing
Webinar Monitoring in era of cloud computingWebinar Monitoring in era of cloud computing
Webinar Monitoring in era of cloud computingCREATE-NET
 

Similar to HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applications Have Done (20)

Where is my MQ message on z/OS?
Where is my MQ message on z/OS?Where is my MQ message on z/OS?
Where is my MQ message on z/OS?
 
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
IBM IMPACT 2014 - AMC-1883 - Where's My Message - Analyze IBM WebSphere MQ Re...
 
Monitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systemsMonitoring and problem determination of your mq distributed systems
Monitoring and problem determination of your mq distributed systems
 
Where is My Message?: Use MQ Tools to Work Out What Applications Have Done
Where is My Message?: Use MQ Tools to Work Out What Applications Have DoneWhere is My Message?: Use MQ Tools to Work Out What Applications Have Done
Where is My Message?: Use MQ Tools to Work Out What Applications Have Done
 
Queue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docxQueue Manager both as sender and Receiver.docx
Queue Manager both as sender and Receiver.docx
 
Websphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentalsWebsphere MQ (MQSeries) fundamentals
Websphere MQ (MQSeries) fundamentals
 
Using Transaction Tracing to Determine Issues with Remote MQ Transactions
Using Transaction Tracing to Determine Issues with Remote MQ TransactionsUsing Transaction Tracing to Determine Issues with Remote MQ Transactions
Using Transaction Tracing to Determine Issues with Remote MQ Transactions
 
MSMQ - Microsoft Message Queueing
MSMQ - Microsoft Message QueueingMSMQ - Microsoft Message Queueing
MSMQ - Microsoft Message Queueing
 
5th KuVS Meeting
5th KuVS Meeting5th KuVS Meeting
5th KuVS Meeting
 
Network-Connected Development with ZeroMQ
Network-Connected Development with ZeroMQNetwork-Connected Development with ZeroMQ
Network-Connected Development with ZeroMQ
 
Functional and non-functional testing with IoT-Testware
Functional and non-functional testing with IoT-TestwareFunctional and non-functional testing with IoT-Testware
Functional and non-functional testing with IoT-Testware
 
Overview of ZeroMQ
Overview of ZeroMQOverview of ZeroMQ
Overview of ZeroMQ
 
Timing attacks against web applications: Are they still practical?
Timing attacks against web applications: Are they still practical?Timing attacks against web applications: Are they still practical?
Timing attacks against web applications: Are they still practical?
 
RTOS implementation
RTOS implementationRTOS implementation
RTOS implementation
 
3450 - Writing and optimising applications for performance in a hybrid messag...
3450 - Writing and optimising applications for performance in a hybrid messag...3450 - Writing and optimising applications for performance in a hybrid messag...
3450 - Writing and optimising applications for performance in a hybrid messag...
 
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
 
dnp3 Protocol Master Client simulator user manual
dnp3 Protocol Master Client simulator user manualdnp3 Protocol Master Client simulator user manual
dnp3 Protocol Master Client simulator user manual
 
WebSphere MQ introduction
WebSphere MQ introductionWebSphere MQ introduction
WebSphere MQ introduction
 
Webinar Monitoring in era of cloud computing
Webinar Monitoring in era of cloud computingWebinar Monitoring in era of cloud computing
Webinar Monitoring in era of cloud computing
 
Cics faqs
Cics faqsCics faqs
Cics faqs
 

More from Matt Leming

533-MigratingYourMQIApplicationsToJMS.pdf
533-MigratingYourMQIApplicationsToJMS.pdf533-MigratingYourMQIApplicationsToJMS.pdf
533-MigratingYourMQIApplicationsToJMS.pdfMatt Leming
 
IBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptxIBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptxMatt Leming
 
Going Deep with MQ
Going Deep with MQGoing Deep with MQ
Going Deep with MQMatt Leming
 
What's new with MQ on z/OS 9.3 and 9.3.1
What's new with MQ on z/OS 9.3 and 9.3.1What's new with MQ on z/OS 9.3 and 9.3.1
What's new with MQ on z/OS 9.3 and 9.3.1Matt Leming
 
Connecting mq&amp;kafka
Connecting mq&amp;kafkaConnecting mq&amp;kafka
Connecting mq&amp;kafkaMatt Leming
 
What's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OSWhat's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OSMatt Leming
 
Building a resilient and scalable solution with IBM MQ on z/OS
Building a resilient and scalable solution with IBM MQ on z/OSBuilding a resilient and scalable solution with IBM MQ on z/OS
Building a resilient and scalable solution with IBM MQ on z/OSMatt Leming
 
What's new in MQ 9.1.* on z/OS
What's new in MQ 9.1.* on z/OSWhat's new in MQ 9.1.* on z/OS
What's new in MQ 9.1.* on z/OSMatt Leming
 
REST APIs and MQ
REST APIs and MQREST APIs and MQ
REST APIs and MQMatt Leming
 
What's new in MQ 9.1 on z/OS
What's new in MQ 9.1 on z/OSWhat's new in MQ 9.1 on z/OS
What's new in MQ 9.1 on z/OSMatt Leming
 
The enterprise differentiator of mq on zos
The enterprise differentiator of mq on zosThe enterprise differentiator of mq on zos
The enterprise differentiator of mq on zosMatt Leming
 
New Tools and Interfaces for Managing IBM MQ
New Tools and Interfaces for Managing IBM MQNew Tools and Interfaces for Managing IBM MQ
New Tools and Interfaces for Managing IBM MQMatt Leming
 
MQ Support for z/OS Connect
MQ Support for z/OS ConnectMQ Support for z/OS Connect
MQ Support for z/OS ConnectMatt Leming
 
HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
 HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen... HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...Matt Leming
 
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
 HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ... HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...Matt Leming
 

More from Matt Leming (15)

533-MigratingYourMQIApplicationsToJMS.pdf
533-MigratingYourMQIApplicationsToJMS.pdf533-MigratingYourMQIApplicationsToJMS.pdf
533-MigratingYourMQIApplicationsToJMS.pdf
 
IBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptxIBM MQ Whats new - up to 9.3.4.pptx
IBM MQ Whats new - up to 9.3.4.pptx
 
Going Deep with MQ
Going Deep with MQGoing Deep with MQ
Going Deep with MQ
 
What's new with MQ on z/OS 9.3 and 9.3.1
What's new with MQ on z/OS 9.3 and 9.3.1What's new with MQ on z/OS 9.3 and 9.3.1
What's new with MQ on z/OS 9.3 and 9.3.1
 
Connecting mq&amp;kafka
Connecting mq&amp;kafkaConnecting mq&amp;kafka
Connecting mq&amp;kafka
 
What's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OSWhat's New In MQ 9.2 on z/OS
What's New In MQ 9.2 on z/OS
 
Building a resilient and scalable solution with IBM MQ on z/OS
Building a resilient and scalable solution with IBM MQ on z/OSBuilding a resilient and scalable solution with IBM MQ on z/OS
Building a resilient and scalable solution with IBM MQ on z/OS
 
What's new in MQ 9.1.* on z/OS
What's new in MQ 9.1.* on z/OSWhat's new in MQ 9.1.* on z/OS
What's new in MQ 9.1.* on z/OS
 
REST APIs and MQ
REST APIs and MQREST APIs and MQ
REST APIs and MQ
 
What's new in MQ 9.1 on z/OS
What's new in MQ 9.1 on z/OSWhat's new in MQ 9.1 on z/OS
What's new in MQ 9.1 on z/OS
 
The enterprise differentiator of mq on zos
The enterprise differentiator of mq on zosThe enterprise differentiator of mq on zos
The enterprise differentiator of mq on zos
 
New Tools and Interfaces for Managing IBM MQ
New Tools and Interfaces for Managing IBM MQNew Tools and Interfaces for Managing IBM MQ
New Tools and Interfaces for Managing IBM MQ
 
MQ Support for z/OS Connect
MQ Support for z/OS ConnectMQ Support for z/OS Connect
MQ Support for z/OS Connect
 
HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
 HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen... HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
HHM-3540: The IBM MQ Light API: From Developer Laptop to Enterprise Data Cen...
 
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
 HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ... HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
HHM-3481: IBM MQ for z/OS: Enhancing Application and Messaging Connectivity ...
 

Recently uploaded

%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisamasabamasaba
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...Health
 
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2
 
Artyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptxArtyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptxAnnaArtyushina1
 
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburgmasabamasaba
 
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdfPayment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdfkalichargn70th171
 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...masabamasaba
 
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrainmasabamasaba
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...masabamasaba
 
Architecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastArchitecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastPapp Krisztián
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisamasabamasaba
 
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyviewmasabamasaba
 
AI & Machine Learning Presentation Template
AI & Machine Learning Presentation TemplateAI & Machine Learning Presentation Template
AI & Machine Learning Presentation TemplatePresentation.STUDIO
 
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfonteinmasabamasaba
 
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Steffen Staab
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...Jittipong Loespradit
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2
 
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...masabamasaba
 

Recently uploaded (20)

%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
 
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
 
Artyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptxArtyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptx
 
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
 
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdfPayment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
 
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain
 
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...
 
Architecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastArchitecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the past
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
 
AI & Machine Learning Presentation Template
AI & Machine Learning Presentation TemplateAI & Machine Learning Presentation Template
AI & Machine Learning Presentation Template
 
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
 
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
 
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
 

HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applications Have Done

  • 1. HHM-2833: Where is My Message?: Using IBM MQ Tools to Work Out What Applications Have Done Matt Leming – lemingma@uk.ibm.com IBM Hursley - UK
  • 2. Symptoms include: • Timeouts for request/response messaging • Inconsistent data between front-end and back-end systems • Customer complaints about ‘missing’ transactions 1
  • 3. Agenda – the MQ Toolbox • Who’s connected? • Are messages flowing? • Where are messages going? • What are the apps doing? • How can I look back in time? Presentation contains MQSC example commands and output. Your own admin tools will vary by task, situation and preference. Tasks can be performed in PowerShell, MQ Explorer etc. as desired. 2
  • 5. DISPLAY CONN DISPLAY CONN(*) TYPE(HANDLE) ALL DIS CONN(*) ALL AMQ8276: Display Connection details. CONN(577C425321295301) EXTCONN(414D5143474154455741593120202020) TYPE(HANDLE) OBJNAME(WLMMDB.REQUEST) OBJTYPE(QUEUE) ASTATE(NONE) HSTATE(INACTIVE) OPENOPTS(MQOO_OUTPUT,MQOO_FAIL_IF_QUIESCING) READA(NO) OBJNAME(SENDINGAPP.REPLY) OBJTYPE(QUEUE) ASTATE(ACTIVE) HSTATE(ACTIVE) OPENOPTS(MQOO_INPUT_SHARED,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING) READA(NO) AMQ8276: Display Connection details. CONN(577C425321295301) EXTCONN(414D5143474154455741593120202020) TYPE(CONN) PID(9740) TID(185) APPLDESC(WebSphere MQ Channel) APPLTAG(jms/GATEWAY1_CF) APPLTYPE(SYSTEM) ASTATE(NONE) CHANNEL(WAS.CLIENTS) CONNAME(127.0.0.1) CONNOPTS(MQCNO_SHARED_BINDING) USERID(pbroad) UOWLOG( ) UOWSTDA(2014-04-08) UOWSTTI(13.24.00) UOWLOGDA( ) UOWLOGTI( ) URTYPE(XA) EXTURID(XA_FORMATID[DSAW] XA_GTRID[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809] XA_BQUAL[00000145414B8AB40000000104DF48FC0001020304050607080900010203040506070809000000010000000000000000 000000000001]) QMURID(0.7940075) UOWSTATE(ACTIVE) Use CONN to match TYPE(CONN) and TYPE(HANDLE) records TYPE(HANDLE) records let you find applications by the objects they access. See all open handles for an app in one place, unlike DIS QSTATUS records Long running UOW information. XID can be tied up with app server txn timeout Channel name + IP help identify client apps. MQ V7.5 and later JMS clients can supply an application name in the CF 4
  • 6. DISPLAY CHSTATUS DISPLAY CHSTATUS(*) ALL AMQ8417: Display Channel Status details. CHANNEL(WAS.CLIENTS) CHLTYPE(SVRCONN) BUFSRCVD(17) BUFSSENT(13) BYTSRCVD(2296) BYTSSENT(2456) CHSTADA(2014-04-08) CHSTATI(15.26.59) COMPHDR(NONE,NONE) COMPMSG(NONE,NONE) COMPRATE(0,0) COMPTIME(0,0) CONNAME(127.0.0.1) CURRENT EXITTIME(0,0) HBINT(5) JOBNAME(0000260C000000B9) LOCLADDR( ) LSTMSGDA(2014-04-08) LSTMSGTI(15.26.59) MCASTAT(RUNNING) MCAUSER(pbroad) MONCHL(OFF) MSGS(6) RAPPLTAG(jar) SSLCERTI(CN=ExampleCA,O=Example) SSLKEYDA( ) SSLKEYTI( ) SSLPEER(SERIALNUMBER=53:43:FD:D6,CN=ExampleApp1,O=Example) SSLRKEYS(0) STATUS(RUNNING) STOPREQ(NO) SUBSTATE(RECEIVE) CURSHCNV(1) MAXSHCNV(1) RVERSION(00000000) RPRODUCT(MQJM) See SSLPEER information not in DIS CONN JOBNAME contains PID (except z/OS): 0x260C = PID(9740) On Linux/UNIX (not Win) TID matches CONN: 0xB9 = TID(185) Note that multiple CONN might share one SVRCONN channel instance Check suitable heartbeats are negotiated 5
  • 7. Automatic detection if no-one is connected
  • 8. AMQSCLM 1) Normal operation Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM Round-robin message distribution App server 1 App server 2 App server N . . . IPPROCS(>=1) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM AMQSCLM AMQSCLM 7
  • 9. AMQSCLM 2) Application failure Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) Messages start building up AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution 8
  • 10. AMQSCLM 3) Failure Detection Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM regularly checks the queue and detects IPPROCS(0) AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution 9
  • 11. AMQSCLM 4) Redirect Messages Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM reduces queue priority in cluster, to stop new messages arriving AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution 10
  • 12. AMQSCLM 5) Un-maroon Msgs (optional) Driver qmgr 1 Driver qmgr 2 Driver qmgr N . . . Target qmgr 1 Target qmgr 2 Target qmgr n . . . Sending appsSending appsSending apps Cluster WLM App server 1 App server 2 . . . IPPROCS(0) IPPROCS(>=1) IPPROCS(>=1) AMQSCLM tells qmgr to redistribute ‘stuck’ messages via the cluster AMQSCLM AMQSCLM AMQSCLM App server N Round-robin message distribution 11
  • 13. AMQSCLM summary • The Cluster Queue Monitoring sample program (AMQSCLM) • Shipped with the product as a sample – Precompiled – Source code – Platforms except on z/OS • More information here: http://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm. mq.dev.doc/q024620_.htm 12
  • 15. Real-time/online Monitoring – Queues • Set detail level for QMGR. Override for individual Queues • Gives live view of application responsiveness ALTER QMGR MONQ(MEDIUM) ALTER QLOCAL(QUEUE1) MONQ(HIGH) ALTER QLOCAL(QUEUE2) MONQ(OFF) AMQ8450: Display queue status details. QUEUE(QUEUE1) TYPE(QUEUE) CURDEPTH(16) IPPROCS(3) LGETDATE(2014-04-08) LGETTIME(17.05.59) LPUTDATE(2014-04-08) LPUTTIME(17.12.16) MEDIALOG( ) MONQ(HIGH) MSGAGE(112) OPPROCS(5) QTIME(10101414, 10101414) UNCOM(NO) DIS QSTATUS(QUEUE1) ALL Age in seconds of the oldest message on the queue Estimations of the time in microseconds that messages are waiting on the queue for processing. First value: Calculated from recent activity Second value: Calculated from longer term activity Timestamps of last PUT/GET to check for recent activity Without MONQ you only get the depth and how many handles are open 14
  • 16. Real-time/online Monitoring – Channels • Gives live view of channel throughput ALTER QMGR MONCHL(MEDIUM) ALTER CHANNEL(CLUSTER1.QM1) CHLTYPE(CLUSRCVR) MONCHL(HIGH) AMQ8417: Display Channel Status details. CHANNEL(MQHUB.GATEWAY2) CHLTYPE(CLUSSDR) BATCHES(52) BATCHSZ(50) BUFSRCVD(55) BUFSSENT(1616) BYTSRCVD(1748) BYTSSENT(1192330) CHSTADA(2014-04-08) CHSTATI(17.49.03) COMPHDR(NONE,NONE) COMPMSG(NONE,NONE) COMPRATE(0,0) COMPTIME(0,0) CONNAME(127.0.0.1(1422)) CURLUWID(0107445310001B34) CURMSGS(50) CURRENT CURSEQNO(11823) EXITTIME(0,0) HBINT(5) INDOUBT(YES) JOBNAME(00002DA4000047D0) LOCLADDR(127.0.0.1(53557)) LONGRTS(999999999) LSTLUWID(0107445310001B33) LSTMSGDA(2014-04-08) LSTMSGTI(17.49.51) LSTSEQNO(11773) MCASTAT(RUNNING) MONCHL(MEDIUM) MSGS(1580) NETTIME(137538,29555) NPMSPEED(FAST) RQMNAME(GATEWAY2) SHORTRTS(180) SSLCERTI( ) SSLKEYDA( ) SSLKEYTI( ) SSLPEER( ) SSLRKEYS(0) STATUS(RUNNING) STOPREQ(NO) SUBSTATE(RECEIVE) XBATCHSZ(20,17) XMITQ(SYSTEM.CLUSTER.TRANSMIT.QUEUE) XQMSGSA(112) XQTIME(545784,3929968) RVERSION(07050002) RPRODUCT(MQMM) DIS CHSTATUS(MQHUB.GATEWAY2) ALL Short/long term calculations of how full your batches are getting, to help you tune BATCHSZ/BATCHINT Last time a message was sent over the channel Depth of messages on XMITQ for this channel (capped at 999) Short/long term calculations of how long messages are waiting on the XMITQ for transmission 15
  • 17. Accounting and Statistics Overview • Monitoring data sent as a PCF message at a configured interval – Statistics – scoped to a Queue / Channel / QMGR – Accounting – scoped to an individual CONN and Queue / QMGR • Applications must consume the messages – Enterprise monitoring – Tivoli ITCAM / Tivoli OMEGAMON XE for Messaging – Sample applications and SupportPacs – Custom applications – com.ibm.mq.headers Java package PCF accounting collection app Queues Channels CONNs PCF statistics collection app Statistics Messages Accounting Messages QMGR SYSTEM.ADMIN.STATISTICS.QUEUE SYSTEM.ADMIN.ACCOUNTING.QUEUE 16
  • 18. Taking a look with SupportPac MS0P • Time period • MQ Statistics at QMGR level • Detailed queue statistics 17
  • 19. Simple practical example using amqsmon • Overall QMGR busyness • Simple data format – Multiple values are [Persistent, NonPersistent] • One message every X seconds – Use amqsmon directly (perl/cron) • Low/high water marks for subscriptions – Grouped by subscription type ALTER QMGR STATMQI(ON) Wait a bit RESET QMGR TYPE(STATISTICS) amqsmon -m GATEWAY1 -t statistics -a -w 0 MonitoringType: MQIStatistics QueueManager: 'GATEWAY1' IntervalStartDate: '2014-04-09' IntervalStartTime: '00.00.35' IntervalEndDate: '2014-04-09' IntervalEndTime: '00.01.13' CommandLevel: 700 ConnCount: 35 PutCount: [271, 0] PutFailCount: 0 Put1Count: [2, 0] Put1FailCount: 0 PutBytes: [273976, 0] GetCount: [270, 0] GetBytes: [269468, 0] GetFailCount: 19 DurableSubscriptionHighWater: [0, 0, 0, 0] DurableSubscriptionLowWater: [0, 0, 0, 0] NonDurableSubscriptionHighWater: [0, 0, 0, 0] NonDurableSubscriptionLowWater: [0, 0, 0, 0] PutTopicCount: [0, 0] PutTopicFailCount: 0 Put1TopicCount: [0, 0] Put1TopicFailCount: 0 PutTopicBytes: [0, 0] PublishMsgCount: [0, 0] PublishMsgBytes: [0, 0] 18
  • 21. Cluster MQ networks can be complex • At each of the dots stuck / mis-sent messages are possible – MQOPENs of the wrong Queue / Queue Manager by apps – Full queues – Stopped channels – Stopped apps – Incorrectly configured QREMOTE/QALIAS routing objects – Cluster membership problems • The standard problem diagnosis approach – Methodically checking channels/queues/DLQs at each point • Is there anything to speed up this process? App1 App2A App2B Requests Replies Replies SDR/RCVR SDR/RCVR 20
  • 22. Cluster Trace Route • MQ has the ability to inject tracer messages – (Can be) hidden from applications – Generate activity reports as they pass through • Tools are available to trace routes using these reports – dspmqrte – command line tool supplied with the product – MS0P – Cat 2 SupportPac extension to MQ Explorer – Report messages need to get back to your ReplyQ of your tool • Lets you see the path messages could have taken – Test connectivity through the MQ network – Test cluster workload balancing • Can quickly jump you close to the problem – The point your trace message veers off in the wrong direction – The point the trail goes cold dspmqrte SDR/RCVR SDR/RCVR dspmqrte dspmqrte 21
  • 23. What are the apps doing?
  • 24. Application Activity Tracing • Similar infrastructure to Accounting & Statistics – PCF messages on SYSTEM.ADMIN.TRACE.ACTIVITY.QUEUE – Configurable via mqat.ini • Can be changed without QMGR restart • Configurable detail level can include partial/full message payload • Frequency options for tuning – Can be enabled on a per-application basis • Via MQCONNX flags • Via application name – In MQ V7.5 and later JMS clients can supply an application name in the connection factory Health warning: Performance impact http://ow.ly/vA8wB 23
  • 25. Application Activity Tracing • Enables scenarios such as – Application audit trail – Message duplication – Resource usage • Which queues or topics are actually being used – Problem Determination • Which Queue / Queue Manager is the application actually opening – Application Coding Standards • Does everyone use the MQI in the recommended way – And more … Health warning: Performance impact http://ow.ly/vA8wB 24
  • 26. Looking at the data with the amqsact sample ALTER QMGR ACTVTRC(ON) <- should be tuned via mqat.ini amqsact -m GATEWAY1 -v MQI Operation: 6 Operation Id: MQXF_PUT ApplicationTid: 12451 OperationDate: '2014-04-09' OperationTime: '01:39:48' High Res Time: 1397003988665548 Completion Code: MQCC_OK Reason Code: 0 Hobj: 18225032 Put Options: 139330 Msg length: 460 Recs_present: 0 Known_dest_count: 1 Unknown_dest_count: 0 Invalid_dest_count: 0 Object_type: MQOT_Q Object_name: 'SENDINGAPP.REPLY' Object_Q_mgr_name: 'GATEWAY1' Resolved_Q_Name: 'SENDINGAPP.REPLY' Resolved_Q_mgr: 'GATEWAY1' Resolved_local_Q_name: 'SENDINGAPP.REPLY' Resolved_local_Q_mgr: 'GATEWAY1' Resolved_type: MQOT_Q Report Options: 0 Msg_type: MQMT_DATAGRAM Expiry: -1 Format_name: 'MQHRF2' Priority: 4 Persistence: 0 Msg_id: 00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 ' 00000010: 0207 4453 2007 2603 '..DS .&. ' Correl_id: 00000000: 414D 5120 4741 5445 5741 5931 2020 2020 'AMQ GATEWAY1 ' 00000010: 0207 4453 2007 2203 '..DS .". ' Reply_to_Q : ' ^D' Reply_to_Q_Mgr: ' ^C' Coded_char_set_id: 1208 Encoding: 273 Put_date: '20140409' Put_time: '00394866' Check the options used for coding standards Check queue name resolution, to find out why messages are going to the wrong place Track individual messages and request/reply scenarios with Msg_id and Correl_id 25
  • 27. Extract from Report MonitoringType: MQI Activity Trace QueueManager: 'V71' Host Name: 'rockall.hursley.ibm.com' CommandLevel: 710 ApplicationName: 'WebSphere MQ Client for Java' ApplicationPid: 18612354 UserId: 'mquser' ConnName: '9.20.95.106' Channel Type: MQCHT_SVRCONN Platform: MQPL_UNIX ======================================================================= = Time Operation CompCode MQRC HObj (ObjName) 10:04:09 MQXF_INQ MQCC_OK 0000 2 10:04:09 MQXF_CLOSE MQCC_OK 0000 2 10:04:09 MQXF_OPEN MQCC_OK 0000 4 () 10:04:09 MQXF_INQ MQCC_OK 0000 4 10:04:09 MQXF_CLOSE MQCC_OK 0000 4 10:04:09 MQXF_OPEN MQCC_OK 0000 4 (SYSTEM.DEFAULT.LOCAL.QUEUE) 10:04:09 MQXF_INQ MQCC_OK 0000 4 26
  • 28. How can I look back in time?
  • 29. What happened to my messages at 2am this morning? • Enterprise monitoring solution – DLQ alerts, queue depth alerts, channel status alerts – Unresolved running units of work – Historical MQ monitoring, accounting and stats data • App logs from the time of the problem – Exceptions, MQ error codes, timeouts • MQ error logs for all qmgrs that could have been involved – Channel errors – Authentication issues • ??? – what else is there 29
  • 30. What about the MQ Recovery Log? • For persistent messages inside transactions – MQ logs each operation performed • Why can't we use this to – Look back in time to 2am and see what happened? – Recovery the original payload if the app lost the message? – See what happened inside long-running units of work? – Provide a list of operations within the failed business transaction? • MQ documents how you can… if – You use the text formatting tool provided with MQ (dmpmqlog) – The logging is linear so the historical data is available in the tool – You follow the right steps to extract data from running qmgrs – You do the work to follow through the logs 30
  • 31. Dmpmqlog output is readable, but analysis is tedious • Wouldn't it be easier to let the computer do the tedious bit? LOG RECORD - LSN <0:0:954:44817> ********** HLG Header: lrecsize 873, version 1, rmid 0, eyecatcher HLRH LogRecdType . . : AQM Put Message (257) Eyecatcher . . : ALRH Version . . . . : 1 LogRecdLen . . : 853 LogRecdOwnr . . : 256 (AQM) XTranid . . . . : TranType: XA XID: formatID 1463898948, gtrid_length 36, bqual_length 54 gtrid [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F2] bqual [000001430B3C84EF000000010000002734721FAD52A950DDA913D08C5C13719A34E164F20000000100000000000 00000000000000001] QueueName . . . : Not known Qid . . . . . . : {Hash 2147211283, Counter: 5} ThisLSN . . . . : <0:0:0:0> PrevLSN . . . . : <0:0:954:43944> Version . . . . : 4 MapIndex . . . : 199 PrevLink.Locn . : 102408 PrevLink.Length : 8 PrevDataLink . : {High 0, Low 103424} Data.Locn . . . : 103424 Data.Length . . : 613 Data . . . . . : 00000: 41 51 52 48 04 00 00 00 FF FF FF FF FF FF FF FF AQRH....ÿÿÿÿÿÿÿÿ 00016: 00 00 00 00 00 00 00 00 C7 00 00 00 02 00 C0 01 ........Ç....... 00032: 00 00 00 00 04 00 01 00 A5 00 00 00 00 00 00 00 ........¥....... 00048: 63 00 00 00 41 4D 51 20 49 49 42 30 31 5F 51 4D c...AMQ IIB01_QM 00064: 20 20 20 20 D9 26 B0 52 20 08 53 F5 30 30 30 30 .&.R .Sõ0000 00080: 30 30 39 39 00 00 00 00 00 00 00 00 00 00 00 00 0099............ 00096: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00112: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................ 00128: 00 00 00 00 FF FF FF FF 00 00 00 00 04 00 00 09 ....ÿÿÿÿ........ 00144: 00 00 00 00 E3 ED 04 80 10 FD 67 E4 FF FF FF FF ....ãí.€.ýgäÿÿÿÿ 00160: 4D 44 20 20 01 00 00 00 00 00 00 00 08 00 00 00 MD ............ 00176: 00 00 00 00 11 01 00 00 B8 04 00 00 4D 51 48 52 ........¸...MQHR 00192: 46 32 20 20 04 00 00 00 01 00 00 00 20 20 20 20 F2 ........ 00208: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00224: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00240: 20 20 20 20 20 20 20 20 20 20 20 20 49 49 42 30 IIB0 00256: 31 5F 51 4D 20 20 20 20 20 20 20 20 20 20 20 20 1_QM 00272: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00288: 20 20 20 20 20 20 20 20 20 20 20 20 4D 55 53 52 MUSR 00304: 5F 4D 51 41 44 4D 49 4E 16 01 05 15 00 00 00 64 _MQADMIN.......d 00320: 20 3E AC 57 48 B3 09 B8 71 B0 4C F2 03 00 00 00 >¬WH..¸q.Lò.... 00336: 00 00 00 00 00 00 00 0B 20 20 20 20 20 20 20 20 ........ 00352: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00368: 20 20 20 20 20 20 20 20 1C 00 00 00 57 65 62 53 ....WebS 00384: 70 68 65 72 65 20 4D 51 20 43 6C 69 65 6E 74 20 phere MQ Client 00400: 66 6F 72 20 4A 61 76 61 32 30 31 33 31 32 31 39 for Java20131219 00416: 31 34 32 32 33 32 32 32 20 20 20 20 00 00 00 00 14223222 .... 00432: A4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ¤............... 00448: 52 46 48 20 00 00 00 02 00 00 00 A4 00 00 01 11 RFH .......¤.... 00464: 00 00 04 B8 4D 51 53 54 52 20 20 20 00 00 00 00 ...¸MQSTR .... 00480: 00 00 04 B8 00 00 00 20 3C 6D 63 64 3E 3C 4D 73 ...¸... <mcd><Ms 00496: 64 3E 6A 6D 73 5F 74 65 78 74 3C 2F 4D 73 64 3E d>jms_text</Msd> 00512: 3C 2F 6D 63 64 3E 20 20 00 00 00 58 3C 6A 6D 73 </mcd> ...X<jms 00528: 3E 3C 44 73 74 3E 71 75 65 75 65 3A 2F 2F 2F 51 ><Dst>queue:///Q 00544: 31 3C 2F 44 73 74 3E 3C 54 6D 73 3E 31 33 38 37 1</Dst><Tms>1387 00560: 34 36 32 39 35 32 32 32 36 3C 2F 54 6D 73 3E 3C 462952226</Tms>< 00576: 43 69 64 3E 30 30 30 30 30 30 39 39 3C 2F 43 69 Cid>00000099</Ci 00592: 64 3E 3C 44 6C 76 3E 32 3C 2F 44 6C 76 3E 3C 2F d><Dlv>2</Dlv></ Ordered unique IDs for each record (LSN) A set of documented record types Transaction information with XIDs, or re-used MQ transaction IDs MQMD header data at discoverable offsets in the hex of a message Put The message payload itself 31
  • 32. Check out dmpmqlog scraper tool • Takes the tedium out of analysing the output from dmpmqlog • Created by Peter Broadhurst • Download from http://www.ibm.com/support/docview.wss?uid=swg21660642 • Generates file per message PUT in the supplied data. • Summary file java -jar dmpmqlog.scraper-20151201.jar -b little-endian -i dmpmqlog.txt -o . 32
  • 33. dmpmqlog scraper tool output • Generates file per message PUT in the supplied data. • Summary file 33
  • 34. Summary • Lots of tools in your MQ toolbox! • On-line Status Commands – DISPLAY CONN – DISPLAY QSTATUS – DISPLAY CHSTATUS • Cluster monitoring – AMQSCLM • Off-line Statistics and Accounting – amqsmon and MS0P to view • Tracking – Trace-route – Application Activity Trace • MQ Recovery Logs – dmpmqlog scraper 34
  • 35. Where can I get more information? IBM Messaging developerWorks developer.ibm.com/messaging IBM Messaging Youtube https://www.youtube.com/IBMmessagingMedia LinkedIn Ibm.biz/ibmmessaging Twitter @IBMMessaging IBM MQ Facebook Facebook.com/IBM-MQ-8304628654/ 35
  • 36. Monday 10:30-11:30 3592 New MQ features 3452 Managing applications 12:00-13:00 2835 MQ on z/OS and Distributed 15:00-16:00 3470 Latest MQ z/OS features 2833 Where is my message? 3544 MQ Light in an MQ infrastructure 16:30-17:30 3573 Hybrid cloud messaging 2941 MQ Advanced Tuesday 08:30-09:30 3540 The MQ Light API 12:00-13:00 3456 The IBM MQ Appliance 13:15-14:15 3499 Introducing Message Hub 3458 MQ Appliance administration 14:30-15:30 6432 MQ updates and futures (InnerCircle) 2849 Messaging feedback roundtable 16:00-17:00 3544 MQ Light in an MQ infrastructure 3513 MQ hands on lab Wednesday 08:30-09:30 3602 Managing your MQ environment 12:00-13:00 3613 Designing MQ self service 6408 Hybrid messaging roadmap (InnerCircle) 13:15-14:00 3416 HA and DR with MQ 3433 Why secure your messaging? 15:45-16:30 3429 Securing MQ 2847 Meet the messaging experts 16:00-17:00 3508 MQ Light hands on lab 16:45-17:30 2275 Migrating to the IBM MQ Appliance Thursday 08:30-09:15 3420 MQ Clustering 2931 Business agility with self service MQ 09:30-10:15 3479 MQ z/OS clusters and shared queue 3450 Optimising MQ applications 2849 Messaging feedback roundtable 10:30-11:15 3465 MQ Appliance high availability 3481 MQ z/OS messaging connectivity 11:30-12:15 3474 Active-active messaging 3537 Monitoring and managing MQ 3425 MQ publish/subscribe Find us at the EXPO: Hybrid Integration peds 65-68 Check out the Hybrid Messaging sub topic under the Hybrid Integration topic for further customer and business partner sessions Hybrid Messaging from the IBM experts at InterConnect 2016 Sunday 14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)
  • 37. MQ as a service Redpaper now available • IBM MQ as a Service: A Practical Approach • Provides information about how to build, deploy, and use IBM MQ as a service. The information in this paper includes the key factors that must be considered while planning the use of IBM MQ as a service • http://ibm.biz/mqaas_red
  • 38.
  • 40. Notices and Disclaimers 40 Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
  • 41. Notices and Disclaimers Con’t. 41 Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
  • 42. Thank You Your Feedback is Important! Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your smartphone, laptop or conference kiosk.