Soumettre la recherche
Mettre en ligne
Crash Dump Analysis 101
•
7 j'aime
•
3,819 vues
John Howard
Suivre
Introduction to illumos Crash Dump Analysis
Lire moins
Lire la suite
Technologie
Industrie automobile
Signaler
Partager
Signaler
Partager
1 sur 32
Recommandé
Crash dump analysis - experience sharing
Crash dump analysis - experience sharing
James Hsieh
Advanced Debugging with WinDbg and SOS
Advanced Debugging with WinDbg and SOS
Sasha Goldshtein
Kernel crashdump
Kernel crashdump
Adrien Mahieux
Linux Crash Dump Capture and Analysis
Linux Crash Dump Capture and Analysis
Paul V. Novarese
Kernel Recipes 2019 - BPF at Facebook
Kernel Recipes 2019 - BPF at Facebook
Anne Nicolas
Linux Kernel Debugging Essentials workshop
Linux Kernel Debugging Essentials workshop
Lubomir Rintel
Kernel_Crash_Dump_Analysis
Kernel_Crash_Dump_Analysis
Buland Singh
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Anne Nicolas
Recommandé
Crash dump analysis - experience sharing
Crash dump analysis - experience sharing
James Hsieh
Advanced Debugging with WinDbg and SOS
Advanced Debugging with WinDbg and SOS
Sasha Goldshtein
Kernel crashdump
Kernel crashdump
Adrien Mahieux
Linux Crash Dump Capture and Analysis
Linux Crash Dump Capture and Analysis
Paul V. Novarese
Kernel Recipes 2019 - BPF at Facebook
Kernel Recipes 2019 - BPF at Facebook
Anne Nicolas
Linux Kernel Debugging Essentials workshop
Linux Kernel Debugging Essentials workshop
Lubomir Rintel
Kernel_Crash_Dump_Analysis
Kernel_Crash_Dump_Analysis
Buland Singh
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Anne Nicolas
Debugging linux kernel tools and techniques
Debugging linux kernel tools and techniques
Satpal Parmar
VS Debugging Tricks
VS Debugging Tricks
Sasha Goldshtein
Kernel Recipes 2019 - Kernel hacking behind closed doors
Kernel Recipes 2019 - Kernel hacking behind closed doors
Anne Nicolas
Proxy arp
Proxy arp
Marian Marinov
Kernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysis
Anne Nicolas
Windows Crash Dump Analysis
Windows Crash Dump Analysis
Microsoft TechNet - Belgium and Luxembourg
SystemV vs systemd
SystemV vs systemd
All Things Open
Introductiontoasp netwindbgdebugging-100506045407-phpapp01
Introductiontoasp netwindbgdebugging-100506045407-phpapp01
Camilo Alvarez Rivera
Android - ADB
Android - ADB
Yossi Gruner
Linux kernel debugging
Linux kernel debugging
libfetion
C++ Production Debugging
C++ Production Debugging
Sasha Goldshtein
Linux kernel debugging(ODP format)
Linux kernel debugging(ODP format)
yang firo
Davide Berardi - Linux hardening and security measures against Memory corruption
Davide Berardi - Linux hardening and security measures against Memory corruption
linuxlab_conf
Systemd cheatsheet
Systemd cheatsheet
Susant Sahani
syzbot and the tale of million kernel bugs
syzbot and the tale of million kernel bugs
Dmitry Vyukov
Debugging linux
Debugging linux
Andrea Righi
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Anne Nicolas
How to Root 10 Million Phones with One Exploit
How to Root 10 Million Phones with One Exploit
Jiahong Fang
Logging system of Android
Logging system of Android
Tetsuyuki Kobayashi
Introduction of unit test on android kernel
Introduction of unit test on android kernel
Johnson Chou
Spark Summit EU talk by Jorg Schad
Spark Summit EU talk by Jorg Schad
Spark Summit
Driver Debugging Basics
Driver Debugging Basics
Bala Subra
Contenu connexe
Tendances
Debugging linux kernel tools and techniques
Debugging linux kernel tools and techniques
Satpal Parmar
VS Debugging Tricks
VS Debugging Tricks
Sasha Goldshtein
Kernel Recipes 2019 - Kernel hacking behind closed doors
Kernel Recipes 2019 - Kernel hacking behind closed doors
Anne Nicolas
Proxy arp
Proxy arp
Marian Marinov
Kernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysis
Anne Nicolas
Windows Crash Dump Analysis
Windows Crash Dump Analysis
Microsoft TechNet - Belgium and Luxembourg
SystemV vs systemd
SystemV vs systemd
All Things Open
Introductiontoasp netwindbgdebugging-100506045407-phpapp01
Introductiontoasp netwindbgdebugging-100506045407-phpapp01
Camilo Alvarez Rivera
Android - ADB
Android - ADB
Yossi Gruner
Linux kernel debugging
Linux kernel debugging
libfetion
C++ Production Debugging
C++ Production Debugging
Sasha Goldshtein
Linux kernel debugging(ODP format)
Linux kernel debugging(ODP format)
yang firo
Davide Berardi - Linux hardening and security measures against Memory corruption
Davide Berardi - Linux hardening and security measures against Memory corruption
linuxlab_conf
Systemd cheatsheet
Systemd cheatsheet
Susant Sahani
syzbot and the tale of million kernel bugs
syzbot and the tale of million kernel bugs
Dmitry Vyukov
Debugging linux
Debugging linux
Andrea Righi
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Anne Nicolas
How to Root 10 Million Phones with One Exploit
How to Root 10 Million Phones with One Exploit
Jiahong Fang
Logging system of Android
Logging system of Android
Tetsuyuki Kobayashi
Introduction of unit test on android kernel
Introduction of unit test on android kernel
Johnson Chou
Tendances
(20)
Debugging linux kernel tools and techniques
Debugging linux kernel tools and techniques
VS Debugging Tricks
VS Debugging Tricks
Kernel Recipes 2019 - Kernel hacking behind closed doors
Kernel Recipes 2019 - Kernel hacking behind closed doors
Proxy arp
Proxy arp
Kernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysis
Windows Crash Dump Analysis
Windows Crash Dump Analysis
SystemV vs systemd
SystemV vs systemd
Introductiontoasp netwindbgdebugging-100506045407-phpapp01
Introductiontoasp netwindbgdebugging-100506045407-phpapp01
Android - ADB
Android - ADB
Linux kernel debugging
Linux kernel debugging
C++ Production Debugging
C++ Production Debugging
Linux kernel debugging(ODP format)
Linux kernel debugging(ODP format)
Davide Berardi - Linux hardening and security measures against Memory corruption
Davide Berardi - Linux hardening and security measures against Memory corruption
Systemd cheatsheet
Systemd cheatsheet
syzbot and the tale of million kernel bugs
syzbot and the tale of million kernel bugs
Debugging linux
Debugging linux
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
How to Root 10 Million Phones with One Exploit
How to Root 10 Million Phones with One Exploit
Logging system of Android
Logging system of Android
Introduction of unit test on android kernel
Introduction of unit test on android kernel
Similaire à Crash Dump Analysis 101
Spark Summit EU talk by Jorg Schad
Spark Summit EU talk by Jorg Schad
Spark Summit
Driver Debugging Basics
Driver Debugging Basics
Bala Subra
Buffer overflow – Smashing The Stack
Buffer overflow – Smashing The Stack
Tomer Zait
Defcon 22-colby-moore-patrick-wardle-synack-drop cam
Defcon 22-colby-moore-patrick-wardle-synack-drop cam
Priyanka Aash
Buffer Overflow - Smashing the Stack
Buffer Overflow - Smashing the Stack
ironSource
sponsorAVAST-VB2014
sponsorAVAST-VB2014
Martin Hron
04 - I love my OS, he protects me (sometimes, in specific circumstances)
04 - I love my OS, he protects me (sometimes, in specific circumstances)
Alexandre Moneger
Csw2016 economou nissim-getting_physical
Csw2016 economou nissim-getting_physical
CanSecWest
02 - Introduction to the cdecl ABI and the x86 stack
02 - Introduction to the cdecl ABI and the x86 stack
Alexandre Moneger
Writing Metasploit Plugins
Writing Metasploit Plugins
amiable_indian
Swug July 2010 - windows debugging by sainath
Swug July 2010 - windows debugging by sainath
Dennis Chung
Simplest-Ownage-Human-Observed… - Routers
Simplest-Ownage-Human-Observed… - Routers
Logicaltrust pl
Filip palian mateuszkocielski. simplest ownage human observed… routers
Filip palian mateuszkocielski. simplest ownage human observed… routers
Yury Chemerkin
07 - Bypassing ASLR, or why X^W matters
07 - Bypassing ASLR, or why X^W matters
Alexandre Moneger
Touch your NetBSD
Touch your NetBSD
Pierre Pronchery
You're Off the Hook: Blinding Security Software
You're Off the Hook: Blinding Security Software
Cylance
Accelerated .NET Memory Dump Analysis training public slides
Accelerated .NET Memory Dump Analysis training public slides
Dmitry Vostokov
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Dmitry Vostokov
Genode Compositions
Genode Compositions
Vasily Sartakov
[2007 CodeEngn Conference 01] dual5651 - Windows 커널단의 후킹
[2007 CodeEngn Conference 01] dual5651 - Windows 커널단의 후킹
GangSeok Lee
Similaire à Crash Dump Analysis 101
(20)
Spark Summit EU talk by Jorg Schad
Spark Summit EU talk by Jorg Schad
Driver Debugging Basics
Driver Debugging Basics
Buffer overflow – Smashing The Stack
Buffer overflow – Smashing The Stack
Defcon 22-colby-moore-patrick-wardle-synack-drop cam
Defcon 22-colby-moore-patrick-wardle-synack-drop cam
Buffer Overflow - Smashing the Stack
Buffer Overflow - Smashing the Stack
sponsorAVAST-VB2014
sponsorAVAST-VB2014
04 - I love my OS, he protects me (sometimes, in specific circumstances)
04 - I love my OS, he protects me (sometimes, in specific circumstances)
Csw2016 economou nissim-getting_physical
Csw2016 economou nissim-getting_physical
02 - Introduction to the cdecl ABI and the x86 stack
02 - Introduction to the cdecl ABI and the x86 stack
Writing Metasploit Plugins
Writing Metasploit Plugins
Swug July 2010 - windows debugging by sainath
Swug July 2010 - windows debugging by sainath
Simplest-Ownage-Human-Observed… - Routers
Simplest-Ownage-Human-Observed… - Routers
Filip palian mateuszkocielski. simplest ownage human observed… routers
Filip palian mateuszkocielski. simplest ownage human observed… routers
07 - Bypassing ASLR, or why X^W matters
07 - Bypassing ASLR, or why X^W matters
Touch your NetBSD
Touch your NetBSD
You're Off the Hook: Blinding Security Software
You're Off the Hook: Blinding Security Software
Accelerated .NET Memory Dump Analysis training public slides
Accelerated .NET Memory Dump Analysis training public slides
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Fundamentals of Complete Crash and Hang Memory Dump Analysis
Genode Compositions
Genode Compositions
[2007 CodeEngn Conference 01] dual5651 - Windows 커널단의 후킹
[2007 CodeEngn Conference 01] dual5651 - Windows 커널단의 후킹
Dernier
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Martijn de Jong
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
debabhi2
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Gabriella Davis
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
Malak Abu Hammad
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
Enterprise Knowledge
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
naman860154
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Rafal Los
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
The Digital Insurer
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
HampshireHUG
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Miguel Araújo
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
Delhi Call girls
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
Antenna Manufacturer Coco
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Radu Cotescu
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
apidays
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
hans926745
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Neo4j
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
The Digital Insurer
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Drew Madelung
Dernier
(20)
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Crash Dump Analysis 101
1.
CRASH DUMP
ANALYSIS 101 JOHN S. HOWARD JOHN.HOWARD@NEXENTA.COM 1 © Copyright Nexenta 2012
2.
AGENDA !
Terminology ! Core Dumps and Crash Dumps ! C Language Basics ! The Mechanism of a Panic ! mdb Overview ! Basic Crash Dump Analysis 2 © Copyright Nexenta 2012
3.
PROCESS, THREAD, LWP !
Process ! A program in execution ! May be comprised of threads or LWPs ! Thread ! The smallest unit of scheduling ! Shared address space and resources ! Light Weight Process (LWP) ! A many-to-1 mapping of user threads to a kernel thread ! Provides user-level multitasking 3 © Copyright Nexenta 2012
4.
INTERRUPTS AND TRAPS !
I nterrupts are asynchronous messages notifying the kernel of external device events ! Some interrupts are handled as traps ! Traps are synchronous messages, essentially a software interrupt ! Bus errors are issued to a processor when referencing a location that can’t be resolved or located 4 © Copyright Nexenta 2012
5.
HANGS, CRASHES, AND
PANICS ! Hang ! Potentially limited or no forensic information ! System up, but unresponsive ! Crash ! Potentially limited forensic information ! System down or rebooted ! Panic ! Maximum potential forensic information ! System down or rebooted 5 © Copyright Nexenta 2012
6.
FORENSIC INFORMATION SOURCES !
Forensic Information Sources ! Console ! syslog, typically logged to /var/adm/messages ! Core file or crash dump 6 © Copyright Nexenta 2012
7.
CORE FILE !
A dump of the contents of all memory allocated to the process ! Inert and static record of state ! Process core files are dumped to the working directory by default ! Core file properties managed via coreadm ! Requires the same libraries to be read 7 © Copyright Nexenta 2012
8.
CRASH DUMP ! A
dump of the contents of all memory allocated to the kernel ! Inert and static record of state ! Written to the pre-specified dump device or swap partition ! Written “backwards” ! Reading requires the same OS version ! Kernel core file facility managed via dumpadm 8 © Copyright Nexenta 2012
9.
DUMPADM !
dumpadm with no options shows current settings # dumpadm !Dump content: kernel pages! !Dump device: /dev/zvol/dsk/rpool/dump (dedicated)! !Savecore directory: /var/crash/myhost! !Savecore enabled: yes! ! To force a crash dump: # savecore -L ! Note that savecore does not quiesce system, so memory contents are changing # uadmin 5 0 # reboot -dn 9 © Copyright Nexenta 2012
10.
PANIC ! Kernel detected
inconsistency ! Protect by exiting ! Three major tasks to be performed in a system panic: ! record information about the panic in memory (making it part of the crash dump) ! synchronize the file systems to preserve user file data ! generate the crash dump 10 © Copyright Nexenta 2012
11.
C PROGRAMMING LANGUAGE
DATATYPES ! Built-ins ! int, float,char ! struct ! A grouping of data ! union ! variant records ! All constituent data items are overlaid ! typedef ! Pointers ! A reference to a memory location 11 © Copyright Nexenta 2012
12.
C DATATYPES EXAMPLES int
ap;! char buf[128];! int *user = sr;! typedef struct smb_mtype {! ! !char! !*mt_name;! ! !int ! !mt_namelen;! ! !int ! !mt_flags;! } smb_mtype_t 12 © Copyright Nexenta 2012
13.
C FUNCTIONS ! Declaration
! Definition ! Parameters are pass by value 13 © Copyright Nexenta 2012
14.
C FUNCTION EXAMPLES Declaration
static void smb_tree_log(smb_request_t *, const char *, ! const char *, ...);! Definition smb_tree_log(smb_request_t *sr, const char *sharename,! const char *fmt, ...) { . . . }! 14 © Copyright Nexenta 2012
15.
PANIC() ! panic(),
cmn_err() ! Common entry points for vpanic() ! Responsible for providing panic information ! die() ! vpanic() ! Assembly language function for saving register state ! ASSERT(condition) ! Halts execution of the kernel if condition is false ! Evaluated and executed only when the DEBUG compilation symbol is defined ! VERIFY(condition) ! Similar to ASSERT, but active even when DEBUG isn’t defined ! Stack will contain assfail() near top 15 © Copyright Nexenta 2012
16.
EXAMPLE 1: PANIC
STRING panic[cpu1]/thread=ffffff000e4e7c60: BAD TRAP: type=e (#pf Page fault) rp=ffffff000e4e77c0 addr=0 occurred in module "unix" due to a NULL pointer dereference 16 © Copyright Nexenta 2012
17.
EXAMPLE 1: STACK
TRACE ffffff000e4e76a0 unix:die+dd () ffffff000e4e77b0 unix:trap+177b () ffffff000e4e77c0 unix:cmntrap+e6 () ffffff000e4e78c0 unix:strcasecmp+16 () ffffff000e4e7a50 smbsrv:smb_tree_log+b3 () ffffff000e4e7a90 smbsrv:smb_tree_connect_core+14a () ffffff000e4e7ac0 smbsrv:smb_tree_connect+35 () ffffff000e4e7ae0 smbsrv:smb_com_tree_connect_andx+16 () ffffff000e4e7b80 smbsrv:smb_dispatch_request+4a9 () ffffff000e4e7bb0 smbsrv:smb_session_worker+6c () ffffff000e4e7c40 genunix:taskq_d_thread+b1 () ffffff000e4e7c50 unix:thread_start+8 () 17 © Copyright Nexenta 2012
18.
MDB – MODULAR
DEBUGGER ! Extensible utility for low-level debugging and editing ! On live kernel: # mdb -k # mdb -kw to edit (VERY DANGEROUS) ! On a core file: mdb syseventd.core.125 ! On a crash dump: # mdb -k unix.3 vmcore.3 18 © Copyright Nexenta 2012
19.
ANALYZE-CRASH.SH ! Extracts the
crash dump from the dump device (savecore -vf filename) if necessary ! Scripted mdb commands for basic crash information: ! Panic string and registers ! dmesg buffer ! Stack ! Thread list ! Executed automatically by the NMC `support` command (NS 3.1.2 and later) 19 © Copyright Nexenta 2012
20.
HAVE I SEEN
THIS BEFORE? ! Footprints ! Known problem or new? ! Redmine ! Search illumos Hg issues https://www.illumos.org/issues/ ! SunSolve is gone, however “We Sun Solve” is rescuing the data from SunSolve.Sun.COM http://wesunsolve.net/bsearch ! illumos Source browser http://src.illumos.org/source/ 20 © Copyright Nexenta 2012
21.
EXAMPLE 1: PANIC
STRING panic[cpu1]/thread=ffffff000e4e7c60: BAD TRAP: type=e (#pf Page fault) rp=ffffff000e4e77c0 addr=0 occurred in module "unix" due to a NULL pointer dereference 21 © Copyright Nexenta 2012
22.
EXAMPLE 1: STACK
TRACE ffffff000e4e76a0 unix:die+dd () ffffff000e4e77b0 unix:trap+177b () ffffff000e4e77c0 unix:cmntrap+e6 () ffffff000e4e78c0 unix:strcasecmp+16 () ffffff000e4e7a50 smbsrv:smb_tree_log+b3 () ffffff000e4e7a90 smbsrv:smb_tree_connect_core+14a () ffffff000e4e7ac0 smbsrv:smb_tree_connect+35 () ffffff000e4e7ae0 smbsrv:smb_com_tree_connect_andx+16 () ffffff000e4e7b80 smbsrv:smb_dispatch_request+4a9 () ffffff000e4e7bb0 smbsrv:smb_session_worker+6c () ffffff000e4e7c40 genunix:taskq_d_thread+b1 () ffffff000e4e7c50 unix:thread_start+8 () 22 © Copyright Nexenta 2012
23.
EXAMPLE 2: PANIC
INFO panic[cpu5]/thread=ffffff000fd72c60: BAD TRAP: type=0 (#de Divide error) rp=ffffff000fd72a40 addr=ffffff02da92e900 sched: #de Divide error addr=0xffffff02da92e900 pid=0, pc=0xfffffffff7ad977b, sp=0xffffff000fd72b30, eflags=0x10246 cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 6f8<xmme,fxsr,pge,mce,pae,pse,de> cr2: fffffd7fff2a60c8 cr3: 5000000 cr8: c rdi: ffffff02d282e840 rsi: 0 rdx: 0 rcx: 64 r8: ffffff000fd72c60 r9: 0 rax: 0 rbx: 0 rbp: ffffff000fd72b90 r10: 0 r11: ffffff02f46e8264 r12: ffffff02da316338 r13: ffffff02da3163d0 r14: ffffff02d5061a50 r15: ffffff02da92e900 fsb: 0 gsb: ffffff02da9a1540 ds: 4b es: 4b fs: 0 gs: 1c3 trp: 0 err: 0 rip: fffffffff7ad977b cs: 30 rfl: 10246 rsp: ffffff000fd72b30 ss: 38 23 © Copyright Nexenta 2012
24.
EXAMPLE 2: STACK
ffffff000fd72920 unix:die+10f () ffffff000fd72a30 unix:trap+1555 () ffffff000fd72a40 unix:cmntrap+e6 () ffffff000fd72b90 cpudrv:cpudrv_monitor+1cb () ffffff000fd72c40 genunix:taskq_thread+285 () ffffff000fd72c50 unix:thread_start+8 () syncing file systems... done dumping to /dev/zvol/dsk/syspool/dump, offset 65536, content: kernel + curproc STACK --- ffffff000fd72b90 cpudrv_monitor+0x1cb(ffffff02da316338) ffffff000fd72c40 taskq_thread+0x285(ffffff02da859140) ffffff000fd72c50 thread_start+8() 24 © Copyright Nexenta 2012
25.
EXAMPLE 2: THREAD
LIST ffffff000fd72c60 fffffffffbc2dbf0 0 0 60 0 PC: panicsys+0x9b TASKQ: cpudrv_cpudrv_monitor stack pointer for thread ffffff000fd72c60: ffffff000fd726e0 xc_insert+0x36() 0xffffff0200000000() cpudrv_monitor+0x1cb() taskq_thread+0x285() thread_start+8() 25 © Copyright Nexenta 2012
26.
EXAMPLE 2: SOURCE CODE From
cpudrv_monitor() 1109 /* 1110 * Adjust counts based on the delay added by timeout and taskq. 1111 */ 1112 idle_cnt = (idle_cnt * cur_spd->quant_cnt) / tick_cnt; 1113 user_cnt = (user_cnt * cur_spd->quant_cnt) / tick_cnt; 1114 26 © Copyright Nexenta 2012
27.
HARDWARE, FIRMWARE, OR
SOFTWARE? ! Crash dumps are inconclusive on hardware errors ! Correlate to fmdump output ! PCI-X panics are the most common hardware caused panic ! PCI Vendor Database http://pcidatabase.com ! KB Article: “Understanding and decoding PCI(-X) Express Fatal Error panics” 27 © Copyright Nexenta 2012
28.
EXAMPLE 3: PANIC
STRING AND STACK TRACE panic[cpu7]/thread=ffffff005cbdbc60: pcieb-3: PCI(-X) Express Fatal Error. (0x101) ffffff005cbdbbb0 pcieb:pcieb_intr_handler+228 () ffffff005cbdbc00 unix:av_dispatch_autovect+7c () ffffff005cbdbc40 unix:dispatch_hardint+33 () ffffff005cbaba80 unix:switch_sp_and_call+13 () ffffff005cbabad0 unix:do_interrupt+b8 () ffffff005cbabae0 unix:_interrupt+b8 () ffffff005cbabbd0 unix:i86_mwait+d () ffffff005cbabc20 unix:cpu_idle_mwait+f1 () ffffff005cbabc40 unix:idle+114 () ffffff005cbabc50 unix:thread_start+8 () 28 © Copyright Nexenta 2012
29.
IDENTIFYING THE PCI-X COMPONENT
Mar 30 2011 00:53:53.606674454 ereport.io.pci.fabric nvlist version: 0 class = ereport.io.pci.fabric ena = 0xbcd565541a801401 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = dev device-path = /pci@0,0/pci8086,3408@1 (end detector) bdf = 0x8 device_id = 0x3408 vendor_id = 0x8086 29 © Copyright Nexenta 2012
30.
IDENTIFYING THE VENDOR
Device ID Chip Description Vendor ID Vendor Name 0x3408 Intel 7500 Chipset PCIe Root Port 0x8086 Intel Corporation device-path = /pci@0,0/pci8086,3408@1 device-path = /pci@0,0/pci8086,3408@1/pci108e,484c@0 device-path = /pci@0,0/pci8086,3408@1/pci108e,484c@0,1 If no entries in neither the PCI vendor database nor `/usr/share/hwdata/pci.ids` then grep `/etc/path_to_inst`: "/pci@0,0/pci8086,3408@1" 0 "pcie_pci" "/pci@0,0/pci8086,3408@1/pci108e,484c@0" 0 "igb" "/pci@0,0/pci8086,3408@1/pci108e,484c@0,1" 1 "igb“ igb is the intel Gigabit NIC driver 30 © Copyright Nexenta 2012
31.
DETERMINE DRIVER AND
PACKAGE DETAILS # dpkg -S igb | grep '/kernel’ sunwigb: /var/lib/dpkg/alien/sunwigb/reloc/kernel/drv/igb.conf sunwigb: /kernel/drv/amd64/igb sunwigb: /var/lib/dpkg/alien/sunwigb/reloc/kernel/drv sunwigb: /kernel/drv/igb sunwigb: /var/lib/dpkg/alien/sunwigb/reloc/kernel Examine the package details: # dpkg -l sunwigb Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Installed/Config-f/Unpacked/Failed-cfg/Half-inst/t-aWait/T-pend |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad) ||/ Name Version Description +++-=======================-======================-====================================== ii sunwigb 5.11.134-31-8234-1 Intel 82575 1Gb PCI Express NIC Driver 31 © Copyright Nexenta 2012
32.
A PCI-X CONCLUSION,
OF SORTS ! Searching redmine for “igb driver” will find a bug, but also check for any Intel 82575 gigabit issues ! Next, determine: ! Is the driver is down revision? ! Is the firmware is down revision? ! If the driver and firmware are current, then this is most likely a hardware problem ! CDA is inconclusive for proving hardware failures 32 © Copyright Nexenta 2012