This presentation will show how to debug Windows virtual machine running on top of open source hypervisor KVM.

1. Windows Guest Debugging
Yan Vugenfirer, yan@daynix.com
Daynix Computing LTD

2. Agenda
Device manager
Logging and DebugView
Introduction to BSOD
Crash dumps
Crash on demand
Driver verifier
WinDbg
Installation and
configuration
Postmortem
debugging
!ndiskd
Configuring guests
Live debugging

3. Device manager

4. Device Manager
Run ->
devmgmt.msc
Devices and drivers
management and
installation
Scan for HW
changes
Effective tool to
check resource
allocations of the
devices, drivers
versions and different
drivers settings.

5. Device Manager - Views
Change the
views of
device
manager
Show non
PNP devices

6. Device Manager - Views
View ->
Devices by
connection
This view
can be used
to check
buses and
devices
relations

7. Device Manager - Views
View -> Resources
by type
Effective to check
IRQ sharing
On OSes below
Vista virtio devices
will use legacy PCI
interrupts
Virtio-serial and
ballon use legacy
interrupts also on
newer OSes.

8. Device Manager - Views
View -> Resources
by connection
IO and memory
address space
with relations
between buses
and the devices

9. Device Manager - Show non-
present devices
Show devices drivers that are installed but the device is curently not present in the
system
Very useful for removing crashing drivers from the system
From command line:
SET devmgr_show_nonpresent_devices=1
START devmgmt.msc

10. Device manager - properties
Double click on the device
Click on “General” tab
You can view device
location on the bus
If the driver is not started
you can see the reason in
status text-box. For
example - no digital
signature on x64 OS from
Vista and up

11. Installing self signed drivers
Drivers need to be at least self signed starting from Vista x64 bit
To enable test signing:
Bcdedit.exe –set TESTSIGNING ON
Disable test signing
Bcdedit.exe –set TESTSIGNING OFF
Install test certificate on the guest
Reboot needed after enabling and disabling test siging commands
More information on test signing:
http://msdn.microsoft.com/en-us/library/bb530195.aspx

12. Device Manager - resources
Double click on the
device
Click on “Resources”
tab
Shows device
resources (IRQs, IO
and memory ranges)

13. Device manager - driver tab
Double click on the
device
Click on “Driver” tab
Get driver version
View the driver
binaries
View the versions of
driver binaires

14. Device manager - details
Double click on the
device
Click on “Details”
tab
A lot of additional
details like: PNP ID
of the device, PM
policy, location of
different device
keys in registry and
etc

15. VirtIO - NetKVM parameters
Advanced tab in device manager
Change logging levels
Enable packet statistics printout
Other parameters starting with
“TestOnly” prefix
Full list: http://www.linux-kvm.org/
page/
Internals_of_NDIS_driver_for_VirtIO
_based_network_adapter#List_of_
configurable_parameters

16. NetKVM - log levels
0 - Basic configuration and unload trace. Critical errors.
1 - Warnings, corner cases
2 - Network packet trace
3 - More verbose trace of packets
4 - Virtio library, DPC
5 - ISR trace
6 - Registers readwrite.

17. NetKVM parameters from
command line
Part of WHQL requirements to supply command line utility for
drivers that expose parameters through “advanced” tab
readme.doc - supplied with driver package explains the usage.
Integrated with “netsh” Windows networking tool
To register (before the first use):
rundll32 netkvmco.dll,RegisterNetKVMNetShHelper
Unregistration (removal)
rundll32 netkvmco.dll,UnregisterNetKVMNetShHelper

18. NetKVM parameters from
command line
Description Command line Output
List devices netsh netkvm show devices
Names and index for each
device
Restart device netsh netkvm restart [idx]
Should be done after
parameters are changed
Show parameters netsh netkvm show parameters [idx]
List the names of the
parameters
Show detailed
information about
parameter
netsh netkvm show paraminfo [idx]
[param]
Current value of the
parameter
Change parameter
value
netsh netkvm setparam [idx] [param]
[value]

19. Debug view and tracing

20. DebugView
Shows debug space of Windows (kind of dmesg).
Can extract debug trace from crash dump if it was
running before the crash
Can log debug trace during boot.
Download from - http://technet.microsoft.com/en-us/
sysinternals/bb896647.aspx

21. DebugView - enable output
to debug space
To enable debug space outputs from Vista and up:
Through registry (using regedit):
Add “Default” value of the type DWORD equal to 0xFFFFFFFF
under “HKEY_LOCAL_MACHINESYSTEMCurrentControlSet
ControlSession ManagerDebug Print Filter”
Reboot
During live debugging
ed nt!Kd_DEFAULT_MASK 0xFFFFFFFF

22. Debug view - capture kernel
Capturing
kernel output
disabled by
default
To enable:
Capture ->
Capture
Kernel

23. Debug view - log boot
Run
DebugView
Enable boot
logging
Run
DebugView
after reboot to
collect the
logs

24. Debug view - crash dump
Run
DebugView
before crash
dump creation
After crash
and reboot -
point
DebugView to
crash location
for processing

25. Virtio debugging with serial port
Uncomment
“ENABLE_COM_DEBUG=1”
in SOURCES file. Comment
out
“ENABLE_PRINT_DEBUG”
Rebuild the driver
On the host side
-serial
tcp::<port>,server,nowait
nc localhost <port>

26. Blue screens

27. Bug check
System crash called “Bug check”
Blue screen
Crash dump will be saved
Ability to debug the bug check if debugger is
connected

28. Blue screen inside
Bug check code
Bug check parameters
- specific to each bug
check
Additional information
- can be located also
below bug check
code and parameters
Crash dump saving counter

29. Common bug checks
Bug Check 0xA: IRQL_NOT_LESS_OR_EQUAL
Bug Check 0x8E: KERNEL_MODE_EXCEPTION_NOT_HANDLED
Bug Check 0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL
Bug Check 0x9F: DRIVER_POWER_STATE_FAILURE
Bug Check 0xC4: DRIVER_VERIFIER_DETECTED_VIOLATION
Bug check codes reference: http://msdn.microsoft.com/en-us/
library/hh994433(v=vs.85).aspx

30. Crash dumps

31. Crash dump creation
During blue screen Windows will save crash dump
Minidumps are not always useful
Also - remove the option to reboot on crash
Right click on “My computer” -> “Properties”->”Advanced
system settings”->”Startup and Recovery” -> “Settings”
Chose “Kernel memory dump”
Remove “Automatically restart”
The windows on different OSes can be different but the
menu names are the same

32. Crash dump creation

33. Crash dump locations
After the crash search in:
%windir%memory.dmp
%windir%MinidumpMinidump<timestamp>.dmp
When you have Windows error message after reboot -
do not close it. Check minidump path and copy it.

34. Crash dumps
From Windows 7 and up - OS will auto-delete large
crash dumps
To keep crash dumps:
Key: HKLMSystemCurrentControlSetControl
CrashControl
Value: “AlwaysKeepMemoryDump”:DWORD set to 1

35. Crash dumps on demand
Crash Windows on demand to examine driver status
Configure through registry:
Key: HKEY_LOCAL_MACHINESystem
CurrentControlSetServicesi8042prtParameters,
Value: CrashOnCtrlScroll, equal to a REG_DWORD
value of 0x01
Key: HKEY_LOCAL_MACHINESystem
CurrentControlSetServiceskbdhidParameters, value:
CrashOnCtrlScroll, equal to a REG_DWORD value of
0x01

36. Crash dumps on demand
Press Ctrl+ScrollLock+ScrollLock
You will get MANUALLY_INITIATED_CRASH
(0x000000E2) bug check.
Read more (also how to redefine the shortcuts): http://
msdn.microsoft.com/en-us/library/ff545499.aspx

37. Crash dumps on demand (2)
Issue NMI from QEMU monitor
Setup guest:
HKLM/System/CurrentControlSet/Control/CrashControl,
DWORD value NMICrashDump 1
Restart
Enter QEMU monitor (Alt+Ctrl+2, to go back Alt+Ctrl+1)
Issue “NMI” command

38. Driver verifier

39. Driver verifier
Built-in into OS (substitutes OS functions with own
layer).
To activate run “verifier.exe” from command line
Need a reboot after the settings are changed
Effective in finding deadlocks, memory leaks, memory
corruptions and incorrect parameters to OS
functions.

40. Driver verifier - activation
Run from
command line:
verifier.exe
Select “Create
custom
settings”

41. Driver verifier - activation

42. Driver verifier - activation
Change
selection to
“Select
driver names
from a list”

43. Driver verifier - activation
Select loaded
driver from the list
Or select driver
from its location on
files system
After clicking finish
- reboot is needed
to activate the
settings.

44. Driver verifier - bug checks
0x000000C4
Some can be
ignored if
debugger is
connected
Provide
additional
information for
debugging

45. Driver verifier - current settings
Run verifier.exe
Click on “Display
existing settings”
radio button
Click on “Next”
button

46. Driver verifier - current settings

47. Driver verifier - removal
Don’t forget to
reboot

48. WinDbg

49. WinDbg
We can examine crash dumps with WinDbg
Remote live kernel debugging
Limited live kernel debugging using one guest only
User mode debugging including remote debugging

50. Installation
Install as part of the WDK
http://msdn.microsoft.com/en-US/windows/
hardware/hh852362
Install as part of the SDK
http://msdn.microsoft.com/en-US/windows/
hardware/hh852363
Download symbols for different Windows OSes
http://msdn.microsoft.com/en-us/windows/
hardware/gg463028

51. Symbols configuration
View OS functions and variable
names
Set Microsoft symbol server and
local paths
Set environment variable:
“_NT_SYMBOL_PATH”
Sample symbol path:
C:symbolslocal;SRV*C:
symbolswebsymbols*http://
msdl.microsoft.com/
download/symbols

52. Symbol configuration
Another way to set the
symbols path is through
WinDbg menu
File ->”Symbol File
Path...”
You can instruct WinDbg
to reload the symbols
Also using windbg
command:
.reload

53. Post mortem debugging
Run
WinDbg.exe
Load dump
file through:
File -> “Open
Crash
Dump...”
Might be slow
first time

54. Loading crash dump
Information
about loaded
symbols
OS information
Hint to use
“!analyze -v”
Bug check
code and
parameters

55. Magic wand - !analyze -v

56. !analyze -v
First command that you should run (or click)
Bug check and its parameters explanation
Hints for the next steps and how those parameters can
be used for further debugging
Module name that probably caused the crash
Stack trace

57. WinDbg - help
Help ->
Contents
Switch to
“Index” tab
for specific
command
help

58. WinDbg - views

59. WinDbg - Views
View To display window
From command
line
Memory Alt+5 d <address>
Call stack Alt+6 kb
Disassembly Alt+7 u <address>
Registers Alt+4 r
Watch Alt+2

60. WinDbg - View memory
Menu: View ->
Memory
Command
(different
variation of “d”):
db <address>

61. Use
case -
memory
leak
with
verifier

62. WinDbg - View disassembly

63. Windbg - Reference symbols
<modulename>!<symbol_name>
Example:
netkvm!DriverEntry

64. WinDbg - Reference symbols

65. Useful command
Command Description
!scsikd Scsi related commands. Use !scsikd.help
to find more
.reload Reload symbols
lm List loaded modules
~<cpu number> Switch between CPUs. For example ~0
!irpfind Search for IO request packets
!irp Display information regarding specific IRP
.reboot Restart remote during live debugging

66. !ndiskd - NDIS debug extensions
!ndiskd.ndis Show ndis.sys build information
!ndiskd.help Show help
!ndiskd.miniports Show NDIS miniports
!ndiskd.miniport <addr> Show info about specific mimiport
!ndiskd.nbl Show NET_BUFFER_LIST info
!ndiskd.oid Show pending OID requests

67. !ndiskd.miniports

68. !ndiskd.miniport <miniport handle>

69. From !ndiskd to NetKVM

70. Configuring live debugging
Host side
Run debugget host VM with : “-serial
tcp::<port_number>,server,nowait”
For example: “-serial tcp::4445,server,nowait”
Run target VM with: “-serial tcp:
127.0.0.1:<port_number>”

71. Configuring live debugging (2)
Host side (new format)
Run debugget host VM with : “-chardev
socket,id=serial0,path=/tmp/debug-
serial0,server,nowait -device isa-
serial,chardev=serial0,id=serial0 ”
Run target VM with: “-chardev
socket,id=serial0,path=/tmp/debug-serial0 -
device isa-serial,chardev=serial0,id=serial0”

72. Configuring live debugging
Guest side - Windows XP and Windows 2003
Edit c:boot.ini
Duplicate the default boot line and add:
“/debug /debugport=COM1 /baudrate=115200”
Reboot

73. Configuring live debugging
Guest side - Vista and up
Use bcdedit.exe
bcdedit /dbgsettings SERIAL DEBUGPORT:1
BAUDRATE:115200
bcdedit /debug ON
Reboot
Read more: http://www.linux-kvm.org/page/
WindowsGuestDrivers/GuestDebugging

74. WinDbg - Connecting to target VM
File -> Kernel
Debug...
Select previously
specified serial
port and baud
rate

75. WinDbg - Breaking into target VM
Ctrl+C
Ctrl+Break
Press pause
button in the
toolbar

76. WinDbg - break points
Command Description
bl List breakpoints
bp Set breakpoint
bu Unresolved breakpoints (will persist between
module reloads)
ba Breakpoint on access
bc Clear breakpoint
Read more: http://msdn.microsoft.com/en-us/library/
windows/hardware/ff552204(v=vs.85).aspx

77. WinDbg - stepping through the code
Command Description
g Start executing the given process or thread
t
Trace - executes single instruction or
source line
p
Step - executes single instruction or source
line. Subroutines are treated as single step.
Check different variation and parameters of the above commands

78. Saving the dump file
Sometimes the system doesn’t save dump file. What to
do?
Connect WinDbg
Use “.dump /f <local_file_name>” to save dump file
Wait for it. It will take a long time

79. Live debugging on same VM
File -> Kernel
Debug...

80. More links
More on debugging - http://msdn.microsoft.com/en-us/library/windows/hardware/
hh406283(v=vs.85).aspx
Use the Microsoft Symbol Server to obtain debug symbol files - http://
support.microsoft.com/kb/311503
Setting break points with WinDbg - http://msdn.microsoft.com/en-us/library/windows/
hardware/ff552204(v=vs.85).aspx
Working with source files - http://msdn.microsoft.com/en-us/library/windows/hardware/
ff556911(v=vs.85).aspx
Live kernel debugging - http://msdn.microsoft.com/en-us/library/windows/hardware/
hh451166(v=vs.85).aspx
Debugging techniques - http://msdn.microsoft.com/en-us/library/windows/hardware/
ff541398(v=vs.85).aspx
Enabling NDIS debug tracing - http://msdn.microsoft.com/en-us/library/windows/
hardware/ff542961%28v=vs.85%29.aspx

81. More tools and links
regedit.exe (just in case)
Sysinternls tools (home of DbgView) - http://
technet.microsoft.com/en-US/sysinternals
Wireshark - http://www.wireshark.org/download.html
Extra debugging tools - http://msdn.microsoft.com/en-
us/library/windows/hardware/ff543998(v=vs.85).aspx

82. Q&A