1. AHMET MİTHAT BOSTANCI MICROSOFT, GTSC IIS/Web Development Support Ahmet.bostanci@microsoft.com Introduction To Debugging ASP.NET applications withWinDBG and Dump Analysis

2. SCOPE This is NOT a full dump analysis workshop. The goal is to introduce the dump analysis as one of the ASP.NET 2.0 web application debugging technics and introduce the tools that help to achieve this task.

3. AGENDA ASP.NET Debugging Technics What is WinDBG When are WinDBG and dump analysis useful? What are dump files? How to collect dump? Basic commands Toolbox & References

4. ASP.NET Debugging Old School Debugging Using Response.Write Mostly not useful in production Not useful in hang scenario Usful in development Useful in production if there is a repro Looks easy but not useful Debugging in VS.NET Good for debugging in development Useful in production with remote debugging / attaching to w3wp.exe Not useful in most production env. issues. Debugging with WinDBG Live debugging / attaching to w3wp.exe Offline debugging: Collecting dumps of w3wp.exe and analysing in WinDBG

6. Debugging with WinDBG Offline debugging Collecting dump files with specific tools for different scenarios Analysing in WinDBG. Live debugging Attaching a process Using breakpoints, catching exceptions, dumping the process out to a file, etc... when necessary.

8. Exceptions and ASP.NET Exceptions are important objects in debugging. FirstChance exceptions Too many exceptions may cause high CPU. Second Chance Exceptions which are not handled Causes crash. First chance exceptions may cause w3wp.exe crashes, if it is thrown in a background thread, such as finalizer threads.

9. When do we need WinDBG? Hang / Performance Problems – Manuel Web site is running slow / performance issues. Web site is leaking memory - ends with performance problems, hang or crash. Web site is running with high memory but not not leaking. Web site is running with 100% CPU. Web site is running with high CPU or CPU usage spikes. Web site is completely hung - Does not response any (dynamic) requests.

10. When do we need WinDBG? Crash Web site or a specific page / method is throwing exceptions but not crashing the w3wp.exe process, first chance exceptions. Web site is crashing - w3wp.exe is crashing. Second chance exceptions.

11. When do we NOT need WinDBG? When there is an easier way for debugging, such as a specific page / method is not functioning as expected In this case, debugging in VS.NET with a repro code might be easier. «Page cannot be displayed» type issues. In this case, troubleshooting IIS or network side is the correct approach.

12. Dump A dump is a snapshot of a process (user mode) or a snapshot of all of the memory in kernel mode. Usermode dump: Process specific, such as w3wp.exe Kernel mode dump: Snapshot of the kernel memory Complete memory dump: Snapshot of the whole memory including kernel and user mode.

13. Types of Dump Minidump Contains only selected parts of the memory used by the process. Fulldump Containsthe full copy of the process memory. Size of the fulldump is somewhat equal to the virtual memory used by the process. This is the dump type we are interested in most of the times.

14. What a dump contains All the things in the memory of the process. Information about the objects in heaps, threads, information about the callstack, stack memory, etc... Limited information about the dead objects which are not garbage collected yet.

15. What a dump does NOT contain Other processes' information. What was the CPU or memory usage in time. It is only a snapshot. Also: Minidump is the smallest in size and does not contain some useful information such as stack objects, local variables, etc...

16. Tools for collecting dumps ADPlus Part of Debugging Tools For Windows Command line adplus -hang -pn w3wp.exe adplus -crash –iis Debug Diagnostic Tool GUI based Easy to configure and use ProcDump Command line Specific for high CPU usage but can be used for collecting manuel dumps as well procdump.exe -c 80 -s 30 -n 5 -ma NNNN >procdump.log Task manager Can be used to collect manuel dump Right click and create userdump Only on Win 7 and Win 2008

17. 32bit vs 64bit If the process is 32bit then the dump should be taken by the32bit version of the related tools. The dump should be opened with the WinDBG’s 32bit version. Same rule applies to 64bit process.

18. Which dump in which scenario? Web site is crashing / throwing exceptions Crash rule with Debug Diagnostic Tool, ADPlus Web site is slow, hanging, using high memory Manuel dump(s) with Debug Diagnostic Tool,ADPlus. Web site is using high CPU or CPU is spiking Manuel dump(s) with Debug Diagnostic Tool, ADPlus. ProcDump for CPU spikes Web site is leaking memory Manuel hang dumps. Memory leak rule created in Debug Diagnostic Tool. Full IIS dumps Manuel dump Collects dumps for all of w3wp.exe, all of dllhost.exe and inetinfo.exe process.

19. How many dump files should I collect? Dump is a snapshot of the memory and does not contain the process history. Most of the crash scenarios One crash dump is enough to start debugging and find the reason of the crash. Performance, hang Consecutive dump files of the same process. For example taking three dumps of w3wp.exe, each are taken one minute apart.

20. Symbol Files Footprint information of functions in EXE or DLL files Necessary to see useful information in dumps, such as callstacks or variables.

21. Private vs Public symbols Private symbols contain local variables, structure type information, functions, source line no., etc... Public symbols are the stripped version of the private symbols.

22. Symbols and WinDBG Symbol server http://msdl.microsoft.com/download/symbols Setting symbol server in WinDBG srv*c:ymbolsublic*http://msdl.microsoft.com/download/symbols Symbols are downloaded to the local folder when needed.

23. Commands in WinDBG WinDBG commands Mostly used in unmanaged debugging. Extensions DLL files Command sets for specific debugging requirements

24. Extensions Copied to the WinDBG installation directory .NET related extensions SOS Comes with .NET framework PSSCOR2 Superset of SOS .NET 2.0 specific extension http://www.microsoft.com/downloads/details.aspx?FamilyID=5c068e9f-ebfe-48a5-8b2f-0ad6ab454ad4&displayLang=en Extract and copy to the WinDBG installation folder. Load from WinDBG: .load psscor2

25. WinDBG and ASP.NET 2.0 dump analysis Open dump file Check and set symbol server Load the PSSCOR2 extenstion .load psscor2

26. Some WinDBG commands Vertarget Shows information about the system on which you are debugging Lm Displays loaded module information ~ Lists all OS threads ~Ns Switch to thread #N K Shows unmanaged call stack of the thread switched. .hh Opens help file .reload /f Forces to reload the symbols. !runaway Shows the thread’s CPU time

27. Some PSSCOR2 commands !DumpAllExceptions (!dae) Shows managed exceptions thrown. !DumpDataTables Shows data tables objects in memory. Useful when debugging memory issues. !DumpColumnNames (!dcn) Prints out the column names for a data table given. !ASPXPages Shows ASPX pages running on all threads. !threadpool Shows ASP.NET thread pool information including the requests in the queue and the current CPU usage of OS. !FindDebugTrue Shows the web applications running in debug mode. !FindDebugModules Shows the modules built in debug mode. !SaveAllModules (!sam) Saves the all modules in the process. Useful when using the reflector to get the code from dump file. !eeversion Prints the .NET framework version. Useful when checking the fixes or service packs installed. !help !help <command>

28. Toolbox Debugging Tools For Windows (WinDBG and ADPlus) http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx Debug Diagnostic Tool v1.1 http://www.microsoft.com/downloads/details.aspx?FamilyID=28BD5941-C458-46F1-B24D-F60151D875A3&displaylang=en ProcDump http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx PSSCOR2 http://www.microsoft.com/downloads/details.aspx?familyid=5C068E9F-EBFE-48A5-8B2F-0AD6AB454AD4&displaylang=en

29. Refernces http://blogs.msdn.com/tess/ (İngilizce) http://blogs.msdn.com/amb/ http://blogs.msdn.com/farukceliktr/ http://blogs.msdn.com/cenkiscan/ http://www.codeproject.com/KB/debug/windbg_part1.aspx http://www.cengizhan.com/

30. Q & A ?