D3 Troubleshooting

1
Rocket® D3 DBMS
Troubleshooting
John Bramley, Senior Director, MV Lab

2
Abstract
 Provide training on how to troubleshoot Rocket® D3 DBMS on
both Windows and UNIX platforms. Focusing on utilities and
commands utilized to troubleshoot D3 in areas of performance,
license management, overflow management, process locking,
D3 server failures and error reporting and management.
©2015 Rocket Software, Inc. All Rights Reserved.

3
Credits and Acknowledgements
Developers
• James Rettally, Senior Software Engineer
• Timothy Rude, Senior Manager, Software Engineering
Co-Presenter
• Mark Locke, Manager, Quality Control

4
Agenda
Performance
License Management
Overflow Management
Lockups (hangs)
Crashes

6
Best Tool Ever – What Changed?
Application
• Application updated or new module added?
• Code changes to a specific subroutine or module?
• Interface added – For example, new web portal utilizing
different interface to D3?
• Additional application load – D3 users added?

7
D3
• Version upgrade or patches installed?
Operating System
• Upgrade or update (for example, Windows updates
applied or Linux packages upgraded)?
• Additional software installed outside of the D3
environment?
• Changes to the network or storage?

8
Virtual Environment
• Additional virtual instance added to the environment?
• Changes made to the resources for the virtual instance
(memory, CPU, network, etc.)?
Physical Hardware
• Upgrade or update?

9
Troubleshooting Tools – D3 TCL
check.pib
d3watcher
watch
buffers (Unix only)

10
Troubleshooting Tools – UNIX Commands
gdb
strace
/proc
top or topas (AIX)
Note – above commands should be run as the ‘root’ user

11
Troubleshooting Tools – Window Commands
Performance Monitor
Resource Monitor
Handle
ProcDump
Process Explorer

13©2015 Rocket Software, Inc. All Rights Reserved.
Performance
What changed?
Identify the bottleneck
Identify the source
Identify the cause

14
Identify the Bottleneck
 CPU
• Windows - nt_pinfo (l, Task Manager Unix - top
 Disk
• Windows - Task Manager Unix – iostat
 Memory
• Windows - nt_pinfo, Task Manager Unix – top, buffers
 Lock Contention
• list-locks, list-lockq
• monitor debugger cs (Windows), gl (Unix)

15
Identify the Source
Single or multiple lines?
Phantom?
External process?
Network?
NAS?
Poorly sized file?
Other?

16
Identify the Cause
Drill down from what was found to be the source
• Program doing loop; for example, system(14) without a sleep
• Extra network traffic added, affecting disk performance
• Data has grown, gradually causing larger memory
requirements
• More users accessing data remotely (new mobile app) adding
to lock contention
D3 problem
• Provide reproducible case, if possible

License Management
What changed?
Is the license count wrong?
Identify the source
Logging

19
Wrong Count?
maxusers
where (az
• +, #, =
d3 –qn (Unix) or %GetNetLicenseInfo()
(Windows)

20
Identify the Source
Is there a pattern?
• Happens once a night?
• Do a lot of kill -9 and/or reset-user?
Types of connections?
• ODBC
• telnet
• MVSP
• RPC

21
Logging
Use the –v option with the License Manager
• Generates a log file
D3 Unix logging
• Use the –v option with parameter
to turn on d3 logging
• Generates several log files

Overflow Management
What changed?
Commands
Logs
Recovery

24
Overflow Management
Is it happening now?
Which process is doing it?
What is the process doing to cause it?

25
Overflow Management - Commands
ovf (at
where
dump <FID> ix
where 1 lz
Ln PCB PIB ABS Stat R1 & Return stack contents 06 Oct 2015
FID Stat Base 11:15:11
*0001 00000081 FF90 000018 0 OVF_ADD:000 BR_STR:000

26
Overflow Management – dump Command
dump 13709 ix
fid: 13709 : 0 0 0 0 ( 358D : 0 0 0 0 )
ARNI 00 0001 00 026A27C8 4426
FM_CREATE_SUB:000 FM_DO_GDM:000 FM_CREATE_FILE:000
ARNI 00 0001 00 030FE871 4425
UPD_REL_DPTR:000 UPD_END:000 FM_DELETE_FILE:000
0000 0000001A 1300B001 74657374 FE333539 0 :........test^359:
0010 34FEFFFF FFFFFFFF FFFFFFFF FFFFFFFF 016 :4^______________:
Most recent check-out: Signature, stat, line, level, time, date
Most recent check-out: Return stack
Most recent check-in: Signature, stat, line, level, time, date
Most recent check-in: Return stack

27
Overflow Management – Logs & Recovery
Logs
• list-errors
• errors,ovf
Recovery
• Full restore
• reclaim-ovf

Lockups
What changed?
Not hung (just super slow)?
Entire server?
All of D3?
Group of D3 users?
A single D3 user?

30
Lockups
Not Hung?
• where
• strace (Unix)
Entire Server?
• Wait, reboot & analyze logs

31
Lockups – All of D3 (Windows)
Generate a process dump
• Task manager
o Find the d3vme.exe
o Right click on it
o Click “Create dump file”
o The location of the dump file will be displayed
• sysinternal.com ProcDump utility
o Must be downloaded and decompressed on system
o procdump –ma d3vme.exe
• Provide dumps for technical support if requested
__

32
Lockups – All of D3 (Unix)
Check semaphores
• d3 –q
• monitor debugger (s command)
strace

33
D3 Lockups – Group of lines
where
Ln PCB PIB ABS Stat R1 & Return stack contents 09 Sep 2015
FID Stat Base 10:14:07
0000 00000080 F310 000018 AU_INPUT:000 AU_START_TCL:000 AU_TCL:000
TCL1:000
0001 00000081 F390 000018 AU_INPUT:000 AU_GET_CMND:000
0002 00000082 BF90 000018 LK_FAIL:000 RET_SRCH:000 BR_READITM:000
BR_READ:000
0003 00000083 BF90 000018 LK_FAIL:000 RET_SRCH:000 BR_READITM:000
BR_READ:000
*0004 00000084 FF90 000018 WS_WHERE1:000 WS_WHATWHERE:000
0256 00000180 BF10 000018 SP_SLEEP:000 SP_SPOOLOUT:000
0513 0000DBA7 BF10 000018 P PP_SCHED:000
list-lockq
Waiting owner fcb group
line# level line# level fid fid item id
0002 00 0001 00 00000A37 00000A28 who
0003 00 0001 00 00000A37 00000A28 who

34
D3 Lockups – A Single Line
where 1 lz
Ln PCB PIB ABS Stat R1 & Return stack contents 08 Oct
2015
FID Stat Base 13:25:
27
*0001 00000081 FF90 000018 0 WS_WHERE1:000 WS_WHATWHERE:000
pibstat ff90
PIB Status ff90
1111 1111 1001 0000
Process is running or activatable
not sleeping
not frame faulted
echoing input
not I/O roadblocked
not reading character
not writing character
not comatized
DCD on
not received XOFF
type-ahead enabled
modem-control enabled
Summary Status:
Process is currently using the CPU ©2015 Rocket Software, Inc. All Rights Reserved.

Crashes
What changed?
Collecting information
Catching it in the act

Crashes
On time crash?
Is there a pattern?
• Everyday at 6PM
• Right after virus scanner runs
Type of connections to D3?
Anything unusual before the crash?

Collecting Information
Errors file
Runtime-errors file
Jobs file
Trace files
Windows Event Log/Unix errpt & perrpt
d3watcher log files
ProcDump/core dump

Catching It in the Act
Install and configure d3watcher
Enable tracing
Download and enable ProcDump (Windows)

40
Summary
Drill down to provide as much information as possible
Call Rocket Technical Support for assistance
Be prepared to answer the question:
What changed?

41
Disclaimer
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED
IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.
IN ADDITION, THIS INFORMATION IS BASED ON ROCKET SOFTWARE’S CURRENT PRODUCT PLANS AND STRATEGY,
WHICH ARE SUBJECT TO CHANGE BY ROCKET SOFTWAREWITHOUT NOTICE.
ROCKET SOFTWARE SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR
OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
• CREATING ANY WARRANTY OR REPRESENTATION FROM ROCKET SOFTWARE(OR ITS AFFILIATES OR ITS OR
THEIR SUPPLIERS AND/OR LICENSORS); OR
• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF
ROCKET SOFTWARE.

42
Trademarks and Acknowledgements
The trademarks and service marks identified in the following list are the exclusive properties of Rocket Software,
Inc. and its subsidiaries (collectively, “Rocket Software”). These marks are registered with the U.S. Patent and
Trademark Office, and may be registered or pending registration in other countries. Not all trademarks owned by
Rocket Software are listed. The absence of a mark from this page neither constitutes a waiver of any intellectual
property rights that Rocket Software has established in its marks nor means that Rocket Software is not owner of
any such marks.
Aldon, CorVu, Dynamic Connect, D3, FlashConnect, Pick, mvBase, mvEnterprise, NetCure,
Rocket, SystemBuilder, U2, U2 Web Development Environment, UniData, UniVerse, and
wIntegrate
Other company, product, and service names mentioned herein may be trademarks or service marks of
others.

D3 Troubleshooting

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

En vedette

En vedette (20)

Similaire à D3 Troubleshooting

Similaire à D3 Troubleshooting (20)

Dernier

Dernier (20)

D3 Troubleshooting