Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne

IMS in a Sysplex Environment:
Challenges and Solutions
© 2014 IBM Corporation
Nick Griffin
World Wide IMS Software Group Sales Advisor
IBM Silicon Valley Lab

Agenda
Product Highlights
IMS Sysplex Review
IMS Sysplex Challenges and Solutions
– Shared Queues
– Data Sharing
– RM Resources
2014
1
Q & A

IMS Sysplex Manager Highlights
– Real-time management of the IMS Sysplex Environment
• Single system image thru local and aggregate view of data
• Simplified User Interface (TSO/ISPF)
• Structured displays of IMS resources and CF structures
• Global Type-1 command, OM Type-2 and IMS SPOC
• Basic z/OS performance information and SVC dump capture
• Statistics for CSL (OM, RM and SCI), IRLM, VSO, and CQS
• Dashboard with key system indicators and threshold monitoring
• Intercept System exceptions and generate Console alerts
2014
2
• Intercept System exceptions and generate Console alerts
• Produce real-time IRLM Long Lock Report
• Browse, delete and recover messages on Shared Queues
• Delete RM resource structure entries
• Assign affinity for transactions in Shared Queues environment
• Protect against buffer overflow in Shared Queues environment
– Support IMS DB/TM, DBCTL, and DCCTL for IMS v11 and later

MVS1
MVS4MVS3
MVS2
CQS1 CQS2
SCI1 RM1 OM1 SCI2 RM2 OM2
IMS Sysplex Manager Sample Configuration
IMSSM
DC
IMSSM
DC
IMS1 IMS2
IMS Connect IMS Connect
FDR1
2014
3
IMSSM
UIS
IMSSM
UIS
FDR1 FDR2

IMS Sysplex Review
2014
4
Source: IMS Redbook (SG24-6908-00)

IMS Sysplex Review - Components
V13
V13
2014
5
Source: IMS Redbook
IRLM
IRLM Lock
V13

IMSplex Configuration
2014
6

Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
Shared Databases
– Long lock detection
2014
7
– Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources
– RM structure content, status, ownership, and removal

Shared Queues
–Transaction affinity
Shared Databases
2014
8
– IRLM statistics
– VSO stats
Shared Resources

Transaction affinity – The Challenges
Force local affinity for efficiency
Assign affinity to specific IMS due to resource availability
Reduce false-scheduling overhead
Reduce contention
2014
9
Reduce contention

Sysplex Manager Tool - Transaction Affinity Highlights
Finer control of transaction scheduling
Non-invasive to existing definition and operation
– No omission of transaction definitions in sysgen
– No stopping of transactions
– No re-classing of dependent regions
– No operational impact for loss of a system
2014
10
User defined affinity to route transaction messages
– Any IMS in the shared queues group
– Any subset of IMS systems
– Equal or weighted distribution

Affinity Routing Statistics – System View
2014
11

Affinity Routing Statistics – Update System
2014
12

Affinity Routing Statistics – Destination View
2014
13

Affinity Routing Statistics – Update Destination
2014
14

Affinity Routing Statistics – Add Destination
2014
15

Shared Queues
–CF structures and log streams
Shared Databases
2014
16
– IRLM statistics
– VSO stats
Shared Resources

IMS Sysplex Review – IMS CF structures and log streams
Log streams
(CQS, OM,
2014
17
(CQS, OM,
RS)

IMS CF Structures and log streams – The Challenges
Growing use of Coupling Facility Structures
– Data Sharing, Shared Message Queues, Resource Manager, CQS log streams
– No single source for list of in use structures and details
Check log stream attributes
2014
18
Monitor offload activities

SM Solution - IMS CF Structures and log streams
Real-time display of structure list
Statistics, Connections, Coupling Facility information
Structure rebuild and checkpoint stats
2014
19
Log streams attributes and offload monitoring

IMS Coupling Facility Structures
2014
20

IMS Coupling Facility Structures
2014
21

Shared VSO structures
2014
22

Shared VSO structures
2014
23

CQS Structures
2014
24

CQS Structures
2014
25

CQS Structures
2014
26

CQS and System Logger
2014
27

SM Solution - CQS Log Stream info
2014
28

SM Dashboard
2014
29

Shared Queues
–Queue monitoring and management
Shared Databases
2014
30
– IRLM statistics
– VSO stats
Shared Resources

IMS Shared Queues
2014
31 Source: IMS Redbook (SG24-6908-00)

Shared Queue Management – The Challenges
Visualize queue content
Browse, Delete, and Recover messages
View CQS processing counters
2014
32

SM Solution - Shared Queue Management
Filter based on Destination name or ID
High level queue depth information, Drill down to levels of detail
Browse, Delete and Recover messages from Cold queue
2014
33
CQS processing counters

Destination Queue Depth
2014
34

2014
35

2014
36

2014
37

2014
38

Shared Queues
–Full Queue Protection
Shared Databases
2014
39
– IRLM statistics
– VSO stats
Shared Resources

Full Queue Protection Challenges
Local Queue
– Local buffer overflow can abend or stall IMS control region
Shared queue structures
– Full structures will stall IMS
2014
40

SM Solution - Full Queue Protection
Local Queue
– Real-time local queue buffer counts and stats
– Reject new messages by employing the Queue Space Notification Exit
(DFSQSSP0)
– Automatic actions for warning, critical or Severe thresholds
– Identify programs using large number of buffers
2014
41
Shared queue structures
– Alert structure utilization exceeding predefined thresholds
– Unable to reject new messages (a possible future enhancement)

Shared Queues
Shared Databases
–Long lock detection
2014
42
–Long lock detection
– IRLM statistics
– VSO stats
Shared Resources

Data Sharing Components
2014
43

Diagnosing Long Locks - The Challenges
DB Lockouts by applications holding IRLM locks for an inordinate amount of
time
– Could go unrecognized until it becomes critical
– Lack of supported tools to assist in recognition and identification of problem
– Manual intervention required to resolve
2014
44

SM Solution - Data Sharing Long Locks
Automatic real-time recognition when IRLM detects
Information consolidated, analyzed for top blocker, and presented
Information recorded in exceptions file and sent to z/OS console
Messages can be sent to z/OS console using user exit so that
automated operations can resolve
2014
45
automated operations can resolve
Problem quickly resolved with automation

Data Sharing Long Lock Exceptions
2014
46

Data Sharing Long Lock Exceptions
2014
47

Shared Queues
Shared Databases
2014
48
–Investigate lock holders and waiters
– IRLM statistics
– VSO stats
Shared Resources

Real-time IRLM / PI Locking Display – The Challenges
Identify resource contention
– Which IMS, dependent region, application is holding resources
– Untangle queue of holder/waiters
– Determine resources with contention
2014
49

SM Solution - Real-time IRLM / PI Locking Display
Display of resources with contention
– Select by dependent region and drill down to resource
– Select by resource and drill down to dependent region
2014
50

Real-time IRLM Locking Display
2014
51

Real-time IRLM Locking Display
2014
52

Dependent Region Display
2014
53

Shared Queues
Shared Databases
2014
54
–IRLM statistics
– VSO stats
Shared Resources

IRLM Statistics – The challenges
Managing the well being of IRLM(s)
– Deadlocks, false contentions, storage utilization?
– Multiple IRLMs to check
2014
55

SM Solution - Aggregated IRLM Statistics
Information gathered from IRLMs across Sysplex
– Aggregated into single system image
– Drill down for information from individual IRLMs
2014
56

Aggregated IRLM Statistics
2014
57

Shared Queues
Shared Databases
2014
58
– IRLM statistics
–VSO stats
Shared Resources

VSO cache hit stats
2014
59

VSO cache hit stats
2014
60

Shared Queues
Shared Databases
2014
61
– IRLM statistics
– VSO stats
Shared Resources
–RM structure entries’ status, ownership, and removal

IMSplex Resource Management
2014
62 Source: IMS Redbook (SG24-6908-00)

IMSplex Resource Management
2014
63

IMS Resource Structure Content
2014
64

Sysplex Serialized Program Management – The Challenges
The IMS who scheduled the serial program abnormally terminated and not
coming back for a while, the scheduled serial program is locked and no
other IMS in the Sysplex can use it.
2014
65

SM Solution - Sysplex Serialized Program Management
Delete the locked RM entry to allow the serial program to be
scheduled by other IMS via ISPF
2014
66

2014
67

2014
68

2014
69

2014
70

2014
71

Summary
Challenges Sysplex Manager
Transaction Affinity management Seamless, persistent, dynamic update of
transaction affinity
CF Structures monitoring All IMS CF info in one place including VSO
Current utilization and checkpoint stats
Log streams offload monitoring Alert of too many offloads
Shared Queues Management Queue depths for destinations and message details
2014
72
Long Lock Detection Real time WTO alerts
Lock Contention report Untangle Holder/Waiter queue for locks
Dependent Region activities
IRLM Statistics report Deadlock and false contention stats
VSO stats Buffer hit stats for shared and non-shared VSO
RM structure management Check ownership and removal of troubled entries

Q & A
2014
73

Other Product Use Scenarios
2014
74

Scenarios
IMS
– Scenario 1 – Taking Inventory and capture diagnostic data
– Scenario 2 – Managing IMS System Parameters
– Scenario 3 – Verifying IMS Resource Definitions
– Scenario 4 – Issuing IMS Commands
– Scenario 5 – Maintaining Command Audit Trail
– Scenario 6 – Managing Dependent Regions
– Scenario 7 – Viewing IMS CF Structures
IMSplex
2014
75
IMSplex
– Scenario 9 – Viewing Aggregate SCI, RM, OM Statistics
Shared Queue and CQS
– Scenario 10 - View local buffer utilization
– Scenario 11 – Managing Destination Queue Depth
– Scenario 12 – Viewing CQS Structures
Dashboard
– Scenario 13 – Dashboard and Thresholds

Scenario 1 – Taking Inventory and Capture Diagnostics
Many address spaces – IMS Control Region, IMS DLI/SAS, IMS
DBRC, IRLM, CQS, RM, OM, SCI, etc..
– How do you identify related IMS components across the Sysplex?
– What is the status of these components?
– What version of IMS components are involved?
– How much resource are they using from z/OS perspective?
– How do you collect diagnostic data to debug sysplex problem?
2014
76
IMS Sysplex Manger structured TSO/ISPF interface
– Guided display of IMS components
– Provides component id, task or job name, version, status and basic z/OS
information such as CPU time and EXCP counts
– Drill-down to detailed component information
– Easily capture console dumps for IMS components across the plex
– Check DBRC RECON datasets placement and VSAM stats

Component List
2014
77

Component List (cont)
2014
78

Capture Console Dumps
2014
79

Capture IMS CF structures Dump
2014
80

z/OS perspective for IMS address spaces
2014
81

DBRC RECON datasets and VSAM stats
Show RECON datasets placement (COPY1, COPY2, SPARE) for all
IMS systems
Built-in LISTCAT for RECONs when drill down
2014
82

DBRC RECON VSAM stats
2014
83

Scenario 2 – Managing IMS System Parameters
Many system run-time parameters
– Sources: DFSPBxxx, overrides via Control Region PARM=
– Which ones are being used?
– Are the parameters the same across the Sysplex?
System parameter display
2014
84
System parameter display
– Real-time scrollable display of “resolved” values
– Parameter values across all IMS systems for easy comparison
– New – System Parameter Tutor for instant description

IMS System Parameters
2014
85

IMS System Parameters
2014
86

IMS System Parameters – Showing Unequal Parms
2014
87

IMS System Parameters – Tutor
2014
88

Scenario 3 – Verifying IMS Resource Definitions
Resource definitions
– Transactions, Programs, Data Bases, Nodes, LTERMS, etc.
– Are the definitions the same across the Sysplex?
– What is the resource status across the Sysplex?
– How do you alter the status across the Sysplex?
Resource definition display
– Query resource by status or attribute
– Real-time scrollable display of resources
2014
89
– Real-time scrollable display of resources
– Resource attributes and status across all IMS systems for easy comparison
– Integrated IMS Type 1 / Type 2 command interface to alter resource status
– Ability to drill-down to related resource (from transaction to PSB, from PSB to
databases… )

IMS Resource Definitions – Filter Specification
2014
90

IMS Resource Definitions – Drill-down to PSB
2014
91

IMS Resource Definitions – Drill-down to Database
2014
92

IMS Resource Definitions – Databases for a PSB
2014
93

IMS Resource Definitions – Manage Transactions
2014
94

IMS Resource Definitions – Manage Transactions
2014
95

Scenario 4 – Issuing IMS Commands
Issue Type 1 command to all IMS systems
– Display same resource type across Sysplex
– Alter resource status across Sysplex
Command issued to each IMS system
– Output recorded to command file and logged history file
2014
96
– Output recorded to command file and logged history file
– ISPF Browse(view) used to display result
– Scrollable, Primary and line commands (find, exclude, etc.)
– Retrievable list of previously entered commands

IMS Commands
2014
97

IMS Commands – Type-1
2014
98

IMS Commands – Type-2
2014
99

Scenario 5 – Maintaining command Audit Trail
Keeping track of operations of your IMS systems
– Need for audit trail?
– Easy way of viewing IMS commands & system messages?
– Automatic archive of audit log?
Sysplex Manager with its history datasets
– Log most type-1 IMS commands, command responses and MTO messages
from all IMS images regardless of origins (z/OS console, terminal, OTMA, AOI
2014
100
from all IMS images regardless of origins (z/OS console, terminal, OTMA, AOI
programs using CMD or ICMD interface)
– Log OM processed command and output
– Searchable, filterable online viewing of log data using ISPF dialog
– Automatic archive for full history dataset
– Archive Directory to assist locating archived data
– Issue type-1 and type-2 IMS commands while viewing log data (similar to z/OS
SDSF)
– Check SM History datasets status

Command Audit Trail
2014
101

Command Audit Trail
2014
102

SM History database status and usage
Up to 8 VSAM Linear datasets make up SM history database
Used in a circular fashion similar to IMS OLDS
Automatic archive process to save full datasets to user-defined output location
Directory to help users keep track of archive output
Real-time information on status of these history datasets
2014
103

View and retrieve archive datasets
Locate which archive dataset has the info you need
Browse the archive dataset instantly
2014
104

Scenario 6 – Managing Dependent Regions
Transaction workload back-logged
– What dependent regions are available?
– What transaction classes do they handle?
– Are they occupied with work?
– What are they doing?
2014
105
Scrollable list of all dependent regions across Sysplex
– Enhanced IMS /DISPLAY ACTIVE REG
– Displays overall DL/I DB/TM call counts
– Sort the lock held column to see which reg holds the most locks
– Resequenced by primary or secondary classes
– Drill down to lower level of detail

2014
106

2014
107

2014
108

2014
109

2014
110

2014
111

Scenarios
IMS
– Scenario 4– Issuing IMS Commands
IMSplex
2014
112
IMSplex
Dashboard

Scenario 8 – Aggregated SCI, RM, OM Statistics
Managing the well being of Common Service Layer (CSL) address
spaces
– Lack of tools to obtain CSL statistical information
– Multiple instances to check
Information gathered from CSL address spaces across Sysplex
2014
113
Information gathered from CSL address spaces across Sysplex
– Aggregated into single system image
– Drill down for information from individual address space

Aggregated RM Statistics
2014
114

2014
115

2014
116

Scenarios
IMS
IMSplex
2014
117
IMSplex
Dashboard

Scenario 9 - SQ local buffer utilization
2014
118

Scenarios
IMS
IMSplex
2014
119
IMSplex
Dashboard

Scenario 12 – Dashboard Sysplex at a Glance
Managing the IMS Sysplex through key system indicators
– Determine overall Sysplex health at a Glance
– Alert when indicators violate predefined thresholds
– Set multiple monitoring thresholds depending on workloads, time of day…
Critical data from key areas summarized on one or more screens
2014
120
Critical data from key areas summarized on one or more screens
– User customized content, positioning, threshold value
– Threshold exceptions are logged for future review
– One or more per user, default supplied
– Automatic screen refresh capability for operational ease
– Drill down for additional detail

Configure Your Dashboard
Create and customize a new dashboard
2014
121

When to monitor
Define monitoring periods
2014
122

Low and High Thresholds
Define threshold values
2014
123

Threshold Processing Interval
Set dashboard data collection interval
2014
124

Dashboard
2014
125

Dashboard – Drill Down
2014
126

References
For publications and demo, visit IMS Sysplex Manager for z/OS at
– http://www-306.ibm.com/software/data/db2imstools/imstools-library.html
2014
127

Thank You!
Your Feedback is Important to Us
2014
128
128

Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (20)

Similaire à Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne

Similaire à Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne (19)

Plus de Robert Hain

Plus de Robert Hain (19)

Dernier

Dernier (20)

Ims09 ims in a sysplex environment - challanges and solutions - IMS UG May 2014 Sydney & Melbourne