Nsd, il tuo compagno di viaggio quando Domino va in crash

Autore: Fabio Pignatti
Professione: Project Manager, Consulente Domino

NSD, il tuo compagno di
viaggio quando Domino va
in crash

http://www.flickr.com/photos/jek-a-go-go/512926428/

Agenda
 Crash e hang, perchè e percome
 NSD, cos'è e perchè serve
 Come decifrare un NSD
 LND come strumento di diagnostica
 Casi reali

3

http://www.flickr.com/photos/donnagrayson/195244498/

Crash e hang, le differenze
 Crash  Hang
• Domino non risponde ai client • Domino non risponde ai client
•
Domino non risponde da console •
Domino risponde da console
• Il sistema operativo è vivo • Il sistema operativo è vivo
 NSD  NSD
 Viene generato un NSD in  Non viene generato un NSD in
automatico automatico ...
 ... ma possiamo generarlo noi!

Le cause
 Il codice delle applicazioni
 Teniamo a bada le teste calde!
 Database corrotti
 Problemi HW (spazio disco, network)
 Software terze parti
 Controllare la compatibilità delle
versioni!

http://www.flickr.com/photos/cayusa/2569878338/

6

http://www.flickr.com/photos/25508895@N00/181902056/

NSD, cos'è e perchè serve (1/2)
 Cos'è?
 NSD = Notes System Diagnostics
 Dove si trova?
 $DOMINO_DATAIBM_TECHNICAL_SUPPORT
 Come si legge?
 È un file ASCII
 È suddiviso in diverse sezioni
 Oggi ci focalizziamo su ...
 NSD7 (da 6.5.5 in poi) ...
 ... ma disponibile anche nelle precedenti

8

NSD, cos'è e perchè serve (2/2)
 Perchè avere un NSD?
 Strategia: dare strumenti di diagnostica per
garantire continuità e affidabilità del servizio
 Altri esempi: DDM, ADC
 Ok ma nel pratico perchè serve NSD?
 Consente di mettere a fuoco le cause del crash e
potenzialmente di risolverle
 Tornando alla stategia ...
 #1233676 “NSD Fix List and NSD Update
Strategy”

http://www.flickr.com/photos/spacepotato/2450300968/

9

Come decifrare un NSD – Le basi (1/4)
 Traccia tutte le info di sistema
 notes.ini
 Processi
 Memoria
 File presenti nella domino directory

11

 Windows (nsd.exe)
 nsd
 nsd -detach (solo W2003 e XP)
 GNU/Linux (nsd.sh)
 nsd
 Alcune opzioni
 nsd -kill
 nsd -batch (solo GNU/Linux)
 nsd -info
 nsd -monitor (solo W32)

12

 Formato del file
 nsd_<Platform>_<ServerName>_YYYY_MM_DD@HH_MM_SS.log
 Ummm ... e questi?
 nsd_all_<Platform>_<Host>_MM_DD@HH_MM.log
 nsd_kill*.log
 nsd_info*.log (GNU/Linux)
 sysinfo*.log (W32)
 console*.log

13


14

Come decifrare un NSD – Process Information (1/3)
 Vengono riportati tutti i processi del sistema
<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>
Section: Notes Process Info (Time 07:34:25)
<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

<@@ ------ Notes Process Info -> Load Modules for Process :: [ nSERVER: 0864] (Time 07:34:28) ------ @@>

Base End Size Module

0x40000000 0x0043c000 245760 C:LotusDominonSERVER.EXE
0x7c800000 0x7c8c0000 786432 C:WINDOWSsystem32ntdll.dll
0x77e40000 0x77f42000 1056768 C:WINDOWSsystem32kernel32.dll

 Nel dettaglio, le chiamate dei processi Domino
<@@ ------ Notes Process Info -> Call Stack for Process :: [ nSERVER: 0864] (Time 07:34:28) ------ @@>

############################################################
### thread 1/144: [ nSERVER: 0864: 0868]
### FP=0006f5a4, PC=7c8285ec, SP=0006f53c
### stkbase=00070000, total stksize=45056, used stksize=2756
############################################################
[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (1388,0,6f5c4,6009668a)
[ 2] 0x77e424ed kernel32.Sleep+15 (1388,0,6f5d8,10003ba0)

15

 Parlando di crash ...
 ... FATAL!
 W32, vengono eseguiti 3 passi con dettaglio crescente
 LINUX, 1 passo
############################################################
### thread 14/144: [ nSERVER: 0864: 08d8] FATAL THREAD (Panic)
### FP=05e7ae30, PC=7c8285ec, SP=05e7adc0
### stkbase=05e80000, total stksize=262144, used stksize=21056
### EAX=0x00000201, EBX=0x77e42004, ECX=0x00000210, EDX=0x00017600
### ESI=0x000003a4, EDI=0x00000000, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00000297
############################################################
############################################################
[ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (3a4,493e0,0,5e7b3b0)
[ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (3a4,493e0,3,5e7b5cc) (Panic) with STACK FRAMES [ nSERVER:
### PASS 2 : FATAL THREAD 0864: 08d8]
############################################################
@[ 3] 0x601a5754 nnotes.OSRunExternalScript@8+1284 (12c,1)
# ---------- Dump of stack from SP to FP ----------
@[ 4] 0x601a5bea nnotes.FRTerminateWindowsResources+986 (1,0,1010,1)
05e7adc0 0B7D827C 1E1DE677 A4030000 00000000 | .}.| ...w .... .... |
@[ 5] 0x601a5faf nnotes.OSFaultCleanupExt@24+895 (f44dd8,1010,0,0,0,5e7b8fc)
@[ 6] 0x601a603a nnotes.OSFaultCleanup@12+26 (0,1010,0) 05e7add0 04AEE705 CCB5E705 10000000 0420E477 | .... .... .... . .w |
05e7ade0 24000000 01000000 00000000 00000000 | $... .... .... .... |
@[ 7] 0x601b12a4 nnotes.OSNTUnhandledExceptionFilter@4+276 (5e7c934)
@[ 8] 0x6017aca8 nnotes.Panic@4+520 (60bb0c80) 05e7adf0 00000000 00000000 00000000 00000000 | .... .... .... .... |
05e7ae00 00000000 00A22F4D FFFFFFFF 00E05DBD | .... ../M .... ..]. |
@[ 9] 0x60002ae3 nnotes.LockHandle@12+323 (2d50,5e7c964,5e7c970)
05e7ae10 04AEE705 00000000 D4ADE705 A0B3E705 | .... .... .... .... |
@[10] 0x6000f3b5 nnotes.OSMemGetSize@8+21 (60eed220,5e7ca34)
@[11] 0x6002b468 nnotes.IDHTableValidate@8+24 (2d50,0) 05e7ae20 DCFFE705 601AE677 481DE677 00000000 | .... `..w H..w .... |
...
@[12] 0x60025d6a nnotes.OSMemIDTableDestructor@4+26 (2d50)
@[13] 0x608ec1e1 nnotes.DbFixup@4+9969 (5e7fb60)
...
16

 Ma sarà utile?
 Aspettiamo il caso reale
############################################################
### PASS 2 : FATAL THREAD with STACK FRAMES 19/20 [ namgr: 4756: 3424]
### FP=0d9ff63c, PC=60651e23, SP=0d9ff630, stksize=12
Exception code: c0000005 (ACCESS_VIOLATION)
############################################################
# ---------- Top of the Stack ----------
# 0d9ff630 00af0000 035a75c0 00000209 0d9ff678 |.....uZ.....x...|
...
@[ 9] 0x62691192 nlsxbe.ANRTItem::ANRTEmbedObject+546 (5ae,0,f3fe74,0,d9ff25c,c003f4)

# 0d9ffa90 0d9ffbe4 62694fc9 000005ae 00000000 |.....Oib........|
# 0d9ffaa0 00f3fe74 00000000 00f3faec 0d326089 |t............`2.|
# 0d9ffab0 0d9ffc64 0d326089 62775b7c 00000003 |d....`2.|[wb....|
# 0d9ffac0 00000000 00f3fe40 00f3faec 0d296aa8 |....@........j).|
# 0d9ffad0 62694ee0 00000014 07cd1e83 0d296aa8 |.Nib.........j).|
# 0d9ffae0 00000000 0d296aa8 0d9ffb2c 0d36608c |.....j).,....`6.|
# 0d9ffaf0 00000001 62694ee0 07cd1ffd 00000000 |.....Nib........|
# 0d9ffb00 00000000 0d9ffb2c 0d36608c 0d296aa8 |....,....`6..j).|
# 0d9ffb10 00000000 0d9ffcb4 00000000 00000088 |................|
# 0d9ffb20 00000001 00000006 0d2e1118 6176614a |............Java|
# 0d9ffb30 746f6c5f 645f7375 6e696d6f 6f6c5f6f |_lotus_domino_lo|
# 0d9ffb40 5f6c6163 68636952 74786554 6d657449 |cal_RichTextItem|
# 0d9ffb50 6d654e5f 4f646562 63656a62 32400074 |_NembedObject.@2|
...

17

Come decifrare un NSD – memcheck (1/5)
 Ricca di informazioni utili: memoria condivisa, db, doc, view
<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>
Section: Notes Memory Analyzer (memcheck) (Time 07:34:58)
<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>
<@@ ------ Notes Memory -> Usage Summary -> Shared Memory Stats :: (Shared) (Time 07:34:59) ------ @@>
TYPE : Count SIZE ALLOC FREE FRAG OVERHEAD %used %free
Static-DPOOL: 131 528482304 510821364 17589916 0 99082 96% 3%
VPOOL : 83 5069008 435807 4488228 0 146381 8% 88%
POOL : 66 3558644 1853232 1657856 0 49676 52% 46%
Overall : 131 528482304 504675280 23736000 0 295139 95% 4%

 MM/OS Structure Information
<@@ ------ Notes Data -> OS Data -> MM/OS Structure Information (Time 07:34:59) ------ @@>
Start Time = 30/06/2008 07.31.40
Crash Time = 30/06/2008 07.34.18
Console Log Enabled = 1
Console Position = 561
Error Message = PANIC: LookupHandle: handle not allocated
Console Position = 561
SharedDPoolSize = 4194304
FaultRecovery = 0x00010010
Cleanup Script Timeout= 300
Crash Limits = 3 crashes in 5 minutes
StaticHang = Virtual Thread [ nSERVER: 0864: 0010] (Native thread [ nSERVER: 0864: 08d8]) (0x864/0x10/0x8d8)
ConfigFileSem = ( SEM:#0:0x010d) n=0, wcnt=-1, Users=-1, Owner=[ : 0000]
FDSem = ( RWSEM:#52:0x410f) rdcnt=-1, refcnt=0 Writer=[ : 0000], n=52, wcnt=-1, Users=0, Owner=[ :
0000]

18


 Open databases
<@@ ------ Notes Data -> NSF Data -> Open Databases (Time 07:34:59) ------ @@>
E:Notesdatadirdb.nsf
Version = 43.0
SizeLimit = 0, WarningThreshold = 0
ReplicaID = 0xc1256fe0:0x004bb680
bContQueue = NSFPool [ 002aa365]
Offline = No
DeleteInProgress = No
FDGHandle = 0xf0241910, RefCnt = 1, Dirty = Y
DB Sem = (FRWSEM:0x0244) state=-1, waiters=0, refcnt=1, nlrdrs=0 Writer=[ nSERVER: 0864: 08b0]
SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1, refcnt=0 Writer=[ : 0000], n=0, wcnt=-1, Users=-1, Owner=[
: 0000]
By: [ nSERVER: 0864: 0006] DBH= 339, User=CN=myserver/O=ACME

19

 Open Documents
<@@ ------ Notes Data -> NSF Data -> Open Documents (BLK_OPENED_NOTE) (numDocs=21, numPools=21) :: [ nevent:
089c] (Time 07:35:00) ------ @@>

DBH NOTEID HANDLE CLASS FLAGS IsProf IsAgent #Pools #Items Size Database
60 2310 0x0024 0x0001 0x0900 No No 1 60 14620 E:Notesdataddm.nsf
...

20

 Open views (globalmente)
<@@ ------ Notes Data -> NIF Data -> NIF Collection Users (hash) (Time 07:34:59) ------ @@>

CollUserVB hColl CollID CollectionVB Remote OFlags ViewNoteID Data HDB/Full View HDB/Full RefCnt SessionID Open
By
------------ ------ ------ ------------ ------ ------ ---------- ------------- ------------- ------ -------------
--------------
[ 0004d005] 115 0 [ 00028005] NO 0x2822 630 140/140 140/140 1 [] [
: 0000]
CollectionGroup:
CurrentCollation = 0
[ 0006c805] 229 0 [ 00033005] NO 0x2822 598 201/201 201/201 1 [] [
: 0000]
CollectionGroup:
[ 00068005] 194 0 [ 00033005] NO 0x2822 598 131/131 131/131 1 [] [
: 0000]
CollectionGroup:
[ 000a6c05] 517 0 [ 00001005] NO 0x2822 270 320/320 320/320 1 []
[ nAMgr: 0c0c: 0004]

21

 Open views (specifico)
<@@ ------ Notes Data -> NIF Data -> NIF Collections (Time 07:34:59) ------ @@>

CollectionVB ViewNoteID UNID OBJID RefCnt Flags Options Corrupt Deleted Temp NS Entries ViewTitle
------------ ---------- -------- ------ ------ ------ -------- ------- ------- ---- --- ------- ------------
[ 00054005] 390 5f8514a1 18930 2 0x4200 00000008 NO NO NO YES 0 ($Profiles)
CIDB = [ 000a7405]
CollSem (FRWSEM:0x030b) state=0, waiters=0, refcnt=0, nlrdrs=0 Writer=[ : 0000]
NumCollations = 1
bCollationBlocks = [ 00053805]
bCollation[0] = [ 00055605]
CollIndex = [ 00000014]
Collation 0:BufferSize 15,Items 1,Flags 0
0: Ascending, by KEY, quot;$0quot;, summary# 1
ResponseIndex [ 02e3fcf0]
NoteIDIndex [ 02e3fcf0]
UNIDIndex [ 02e3fcf0]

22

Come decifrare un NSD – resource usage summary
 La sezione inzia con
<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>
Section: Notes Memory Analyzer (memcheck) -> Resource Usage Summary (Time 07:35:01)
<@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>

 VThread
** VThread [ nAMgr: 0bec: 0002]
.Mapped To: PThread [ nAMgr: 0bec: 0bf0]
.. SOBJ: addr=0x0124926c, h=0xf010403d t=0xc30a (BLK_LOOKUP_THREAD)
.. SOBJ: addr=0x01235fdc, h=0xf0104016 t=0xc176 (BLK_SDKT)
.. SOBJ: addr=0x01247774, h=0xf010403b t=0xc275 (BLK_NSFT)
.. SOBJ: addr=0x01202d0c, h=0xf0104001 t=0xc130 (BLK_TLA)
.. SOBJ: addr=0x012362f4, h=0xf0104018 t=0xc436 (BLK_LSITLS)
.. Database: E:Notesdatadatabase.nsf
.... DBH: 593, By: CN=myserver/O=ACME, WasAccessed=Yes
.. Database: E:Notesdatadatabase2.nsf
.... DBH: 498, By: CN=myserver/O=ACME, WasAccessed=Yes
...... doc: HDB= 498, NoteID= 770, hNote=0x0010, flags=0000, class=0200 [Agent Name=quot;EliminaPaquot;]
.... DBH: 563, By: CN=Mario Rossi/O=ACME, WasAccessed=Yes
.... DBH: 475, By: CN=Mario Rossi/O=ACME, WasAccessed=Yes
...... doc: HDB= 475, NoteID=2294, hNote=0x2d60, flags=0200, class=0001
...... doc: HDB= 475, NoteID= 770, hNote=0x0016, flags=0000, class=0200 [Agent Name=quot;EliminaPaquot;]
.. file: fd: 920, E:Notesdatadatabas2.nsf
.. file: fd: 1112, E:Notesdatadatabase.nsf
.. file: fd: 396, E:NotesdataIBM_TECHNICAL_SUPPORTconsole.log

23

http://www.flickr.com/photos/71502646@N00/2412677222/

LND come strumento di diagnostica (1/3)
 Lotus Notes Diagnostic
 È un database Notes integrato con il sistema operativo (W32)
 2.6 alla data odierna
 http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg24019151

25

 Come si usa

26

 Cosa si ottiene?
 Filtra l'NSD con le info più significative
 Permette di navigare in modo “visuale” un NSD
 Si collega alla KB IBM ed esegue per noi una ricerca!
 Consente di copiare il thread con il fatal in modo veloce
 Permette di confrontare vari NSD

27

http://www.flickr.com/photos/doctorow/2496308570/

Un caso reale: è lento (1/2)
 Problema
 Drrriiiiiiiiiiin!
 “È lento! Aiuto!”
 Ma sono passato la settimana scorsa e mi pareva performante ...
 No, nessun cambiamento ... (ndr è in piedi da 2 gg la nuova procedura)
 Si ma “quanto” è urgente?
 MSPI

29

Un caso reale: è lento (2/2)
 Generiamo un NSD durante il rallentamento
 Nsd -detach (2003)
 Analisi
 LND, nessun crash
 Ma aprendo il file ...
** VThread [ tomcat5: 1636: 5]
.Mapped To: PThread [ tomcat5: 1636: 4484]
.. SOBJ: addr=0x0ff83620, h=0xf010402a t=0xd1ba (PKG_LSBE+10)
.. SOBJ: addr=0x0ff834a0, h=0xf010402b t=0xc130 (BLK_TLA)
.. SOBJ: addr=0x00748b6c, h=0xf010402c t=0xc30a (BLK_LOOKUP_THREAD)
.. SOBJ: addr=0x0ff83640, h=0xf010402d t=0xc275 (BLK_NSFT)
.. Database: D:LotusDominoDatadb.nsf
.... DBH: 304, By: Anonymous
...... doc: HDB=304, ID=2794, H=28, class=0001, flags=0101
...... view: hCol=352, cg=N, noteID=302, someView|someView
...... doc: HDB=269, ID=2794, H=31, class=0001, flags=0101
 Soluzione

30

Tranquilli c'è Jamie!

 Risoluzione dei problemi di prestazioni di applicazioni Lotus Domino

 Jamie Magee
 In sala Michelangelo dalle 15:20 alle 16:00

31

http://www.flickr.com/photos/aspengull/111255602/

Un caso reale: è giù (1/4)
 Problema
 Drrriiiiiiiiiiin!
 “È giù!”
 ...
 MSPI

33

 Domino opportunamente configurato
 Nsd generato in automatico
 Analisi, LND
 Crash rilevato. Stringhe di ricerca per la KB ma ...

34

 ... nessun risultato trovato

 Raffinando la ricerca:
 SPR# BRIS5TXQJU - Fixed an HTTP crash during image conversion
under certain circumstances. This problem rarely occurs in 6.5x but
code changes make it more likely in 7.0.2 FP1.
 SPR# COLY6Y2J3Q - Fixed an HTTP page rendering crash. This
regression was introduced in 7.0.2 FP1.

35

 Analisi, NSD
############################################################
### FATAL THREAD 98/106 [ nhttp: 0270: 11c0]
### FP=0x4a4ba914, PC=0x60004141, SP=0x4a4ba910
### stkbase=4a4c0000, total stksize=262144, used stksize=22256
### EAX=0x4a4ba9d8, EBX=0x4a8e0b58, ECX=0x4a8e0b58, EDX=0x131182dc
### ESI=0x00000002, EDI=0x4a4baa84, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202
############################################################
...
############################################################
### PASS 2 : FATAL THREAD with STACK FRAMES 98/106 [ nhttp: 0270: 11c0]
### FP=4a4ba914, PC=60004141, SP=4a4ba910
### stkbase=4a4c0000, total stksize=262144, used stksize=22256
############################################################
...
4a4bab60 8C09A426 01000000 8C09A426 4CDF4B4A | ...& .... ...& L.KJ |
4a4bab70 34820310 FFFFFFFF 88AD4B4A FB070210 | 4... .... ..KJ .... |
4a4bab80 8010A426 8C09A426 436F6E74 656E742D | ...& ...& Cont ent- |
4a4bab90 54797065 3A20696D 6167652F 67696600 | Type : im age/ gif. |
4a4baba0 20636861 72736574 3D49534F 2D383835 | cha rset =ISO -885 |
4a4babb0 392D3100 70020000 01000000 7CAB4B4A | 9-1. p... .... |.KJ | http://www.flickr.com/photos/blude/2665906010/
4a4babc0 14B07549 3A000000 8010A426 8010A426 | ..uI :... ...& ...& |
 Soluzione
36

http://www.flickr.com/photos/razielss/1632964902/

Bonus track!
 Test degli applicativi
 ... non solo Domino!
 Software terze parti
 Manutenzione periodica degli archivi
 Transaction logging
 Censimento dei sistemi
 ... non solo Domino!
 Alcune accortezze che non guastano
 Unread marks
 Indici FT
 Quota

 KIS, keep it simple

38

Nsd, il tuo compagno di viaggio quando Domino va in crash

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (18)

Similaire à Nsd, il tuo compagno di viaggio quando Domino va in crash

Similaire à Nsd, il tuo compagno di viaggio quando Domino va in crash (20)

Dernier

Dernier (20)

Nsd, il tuo compagno di viaggio quando Domino va in crash