Analyzing Windows Kernel Dumps

This post contains information on steps I took to investigate a kernel mode dump with a repeatable 0x7F bugcheck caused on a workstation when printing a specific document to a generic LaserJet 4000 on a remote print server.

After analysing the kernel dump, the problem appeared to be occurring in the kernel-mode GDI win32k.sys driver, and using this information I was able to find a hotfix that did resolve the issue.

I don't pretend to understand a large portion of what was visible in the kanalyze results, but the information that I did glean was enough to point to the hotfix very quickly.

The following steps were taken to analyse the kernel dump created by the stop 0x7F error:

1. Install debugging tools, The Kanalyze guide recommends version 6.5.3.8 32-bit (the most recent is dbg_x86_6.9.3.113.msi)
2. Install Kanalyze 8.1. The default directory is c:\kktools\kanalyze8.1
3. Copy c:\kktools\kanalyze8.1\x86\*.* to the ‘\Program Files\Debugging Tools for Windows’ directory (including the plug-ins subdirectories)
4. Run ‘\Program Files\Debugging Tools for Windows\kanalyze.exe
   
   1. Perform a new memory dump
      
   2. Type the location of the dump file (usually c:\windows\memory.dmp if a local dump)
      
   3. Leave the symbol and binaries path to the HTTP locations (requires Internet access)
      
   4. Select ‘Show Detail’ and then Next
   
   
   5. The crashdump analysis will begin, which can be relatively time consuming (~30 minutes for the small 87MB crashdump of my workstation).
   
   
5. When the analysis is complete, select ‘Browse’ to view the results, including
   
   1. Analysis Summary – The automatically generated summary, providing a best-guess of the potential cause of the crash. Note that in this crash, the AnalysisSummary.txt file shows that the best guess was rdbss.sys – the redirected drive buffering subsystem, used by the SMB mini-redirector.
      
   2. Analysis TriageInfo CrashStackInfo – Information on the kernel stack at the time of the crash. For this crash, this information seems more useful, as it shows the GDI bitblt and dithercolor functions being called, followed by a divide by zero trap in the kernel and the bugcheck. This gives quite a bit more information to go on when searching for cause and resolution
      
   3. Analysis TriageInfo CrashProcess – The process that caused the crash, in this case, winword.exe, providing information on the memory and processed (useful for other areas in the debug namespace)
      
   4. Analysis TriageInfo CrashThread – The thread that caused the crash, in this case, the kernel thread ID and start address space, useful for other areas in the debug namespace
      
   5. Analysis Deadlock – Current deadlocks – conflicts between threads trying to exclusively access code. Locks occur constantly, but a deadlock can occur when two or more threads have code locked and are waiting for the other(s) to continue. The deadlock in this scenario is presumed to be a result of the divide by zero kernel error, and not causative
      
   6. Root – Provides an interesting overview of the memory allocation of the system, including boot, HAL, non-paged/paged, kernel and usermode address spaces.
   
   

The following stack information from the crash dump was what pointed me in the hotfix direction, the bitblt and drvdither calls which resulted in a divide by zero error in the kernel:

          +------------+
          |            |   StackLimit   = aa1f5000
          +------------+
          |      :     |
          +------------+
          |            |   KernelStack  = aa1f85d8
          +------------+
   /|     |            |
    |     |      :     |
    |     |            |
  stack   +------------+
  growth  |            |   InitialStack = aa1f9000
          +------------+

ChildEBP Ret-Addr Called Procedure
---------------------------------------------------------------------------
aa1f8984 80596813 nt!KeBugCheck + 14
aa1f89dc 8053d60f nt!Ki386CheckDivideByZeroTrap + 41
aa1f89dc bf903230 nt!KiTrap00 + 83
aa1f8a94 bf81af4d win32k!UMPDDrvDitherColor + 7d
aa1f8b24 bf826e70 win32k!bGetRealizedBrush + 2bd
aa1f8b3c bf82f869 win32k!pvGetEngRbrush + 1f
aa1f8b98 bf805363 win32k!EngBitBlt + 283
aa1f8be8 bf80bd41 win32k!GrePatBltLockedDC + 1ea
aa1f8d54 8053c9fa win32k!NtGdiFlushUserBatch + 689
aa1f8d70 aaea754a nt!KiFastCallEntry + ca
aa1f8ddc 805411c2 rdbss!RxpWorkerThreadDispatcher + 18a
aa1f8de8 00000000 nt!KiThreadStartup + 16

Notes:

1. The analysis requires more than a minidump. Use sysdm.cpl Advanced Startup and Recovery Settings to write at least a ‘Kernel memory dump’, rather than a small / minidump.
2. The console can be used to execute plug-in commands, or standard kernel debug commands (eg VIEWCMD or kd !ntsdexts.locks). Either click on the console button, or use the File Command Window menu option on the browser

References

Debugging Tools – v6.5.3.8:
http://msdl.microsoft.com/download/symbols/debuggers/dbg_x86_6.5.3.8.exe

Kernel Memory Space Analyzer Version 8.1
http://www.microsoft.com/Downloads/details.aspx?FamilyID=e84d3b35-63c3-445b-810d-9fed3fdeb13f&displaylang=en

First Step Guide to Kernel Memory Space Analyzer V8.1 (Kanalyze)
http://download.microsoft.com/download/8/3/5/835efd1b-d801-4aa5-84ab-c4809fb493f4/Kanalyze%20First%20Step%20Guide%20V8.1.ppt

BitBlt
http://msdn.microsoft.com/en-us/library/ms532278(VS.85).aspx

DrvDitherColor
http://msdn.microsoft.com/en-us/library/ms793395.aspx

Debugging Deadlocks (No Ready Threads)
http://msdn.microsoft.com/en-us/library/cc267852.aspx

Error message when you try to print after you install the GDI security update 925902: "Stop 0x0000007F"
http://support.microsoft.com/kb/935843

Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.