Labels

Tuesday, July 15, 2008

Analyzing Windows Kernel Dumps

This post contains information on steps I took to investigate a kernel mode dump with a repeatable 0x7F bugcheck caused on a workstation when printing a specific document to a generic LaserJet 4000 on a remote print server.

After analysing the kernel dump, the problem appeared to be occurring in the kernel-mode GDI win32k.sys driver, and using this information I was able to find a hotfix that did resolve the issue.

I don't pretend to understand a large portion of what was visible in the kanalyze results, but the information that I did glean was enough to point to the hotfix very quickly.

The following steps were taken to analyse the kernel dump created by the stop 0x7F error:

  1. Install debugging tools, The Kanalyze guide recommends version 6.5.3.8 32-bit (the most recent is dbg_x86_6.9.3.113.msi)
  2. Install Kanalyze 8.1. The default directory is c:\kktools\kanalyze8.1
  3. Copy c:\kktools\kanalyze8.1\x86\*.* to the ‘\Program Files\Debugging Tools for Windows’ directory (including the plug-ins subdirectories)
  4. Run ‘\Program Files\Debugging Tools for Windows\kanalyze.exe
    1. Perform a new memory dump
    2. Type the location of the dump file (usually c:\windows\memory.dmp if a local dump)
    3. Leave the symbol and binaries path to the HTTP locations (requires Internet access)
    4. Select ‘Show Detail’ and then Next

    5. The crashdump analysis will begin, which can be relatively time consuming (~30 minutes for the small 87MB crashdump of my workstation).

  5. When the analysis is complete, select ‘Browse’ to view the results, including
    1. Analysis Summary – The automatically generated summary, providing a best-guess of the potential cause of the crash. Note that in this crash, the AnalysisSummary.txt file shows that the best guess was rdbss.sys – the redirected drive buffering subsystem, used by the SMB mini-redirector.
    2. Analysis TriageInfo CrashStackInfo – Information on the kernel stack at the time of the crash. For this crash, this information seems more useful, as it shows the GDI bitblt and dithercolor functions being called, followed by a divide by zero trap in the kernel and the bugcheck. This gives quite a bit more information to go on when searching for cause and resolution
    3. Analysis TriageInfo CrashProcess – The process that caused the crash, in this case, winword.exe, providing information on the memory and processed (useful for other areas in the debug namespace)
    4. Analysis TriageInfo CrashThread – The thread that caused the crash, in this case, the kernel thread ID and start address space, useful for other areas in the debug namespace
    5. Analysis Deadlock – Current deadlocks – conflicts between threads trying to exclusively access code. Locks occur constantly, but a deadlock can occur when two or more threads have code locked and are waiting for the other(s) to continue. The deadlock in this scenario is presumed to be a result of the divide by zero kernel error, and not causative
    6. Root – Provides an interesting overview of the memory allocation of the system, including boot, HAL, non-paged/paged, kernel and usermode address spaces.

The following stack information from the crash dump was what pointed me in the hotfix direction, the bitblt and drvdither calls which resulted in a divide by zero error in the kernel:



          +------------+
          |            |   StackLimit   = aa1f5000
          +------------+
          |      :     |
          +------------+
          |            |   KernelStack  = aa1f85d8
          +------------+
   /|     |            |
    |     |      :     |
    |     |            |
  stack   +------------+
  growth  |            |   InitialStack = aa1f9000
          +------------+

ChildEBP Ret-Addr Called Procedure
---------------------------------------------------------------------------
aa1f8984 80596813 nt!KeBugCheck + 14
aa1f89dc 8053d60f nt!Ki386CheckDivideByZeroTrap + 41
aa1f89dc bf903230 nt!KiTrap00 + 83
aa1f8a94 bf81af4d win32k!UMPDDrvDitherColor + 7d
aa1f8b24 bf826e70 win32k!bGetRealizedBrush + 2bd
aa1f8b3c bf82f869 win32k!pvGetEngRbrush + 1f
aa1f8b98 bf805363 win32k!EngBitBlt + 283
aa1f8be8 bf80bd41 win32k!GrePatBltLockedDC + 1ea
aa1f8d54 8053c9fa win32k!NtGdiFlushUserBatch + 689
aa1f8d70 aaea754a nt!KiFastCallEntry + ca
aa1f8ddc 805411c2 rdbss!RxpWorkerThreadDispatcher + 18a
aa1f8de8 00000000 nt!KiThreadStartup + 16

Notes:

  1. The analysis requires more than a minidump. Use sysdm.cpl Advanced Startup and Recovery Settings to write at least a ‘Kernel memory dump’, rather than a small / minidump.
  2. The console can be used to execute plug-in commands, or standard kernel debug commands (eg VIEWCMD or kd !ntsdexts.locks). Either click on the console button, or use the File Command Window menu option on the browser

References

Debugging Tools – v6.5.3.8:
http://msdl.microsoft.com/download/symbols/debuggers/dbg_x86_6.5.3.8.exe

Kernel Memory Space Analyzer Version 8.1
http://www.microsoft.com/Downloads/details.aspx?FamilyID=e84d3b35-63c3-445b-810d-9fed3fdeb13f&displaylang=en

First Step Guide to Kernel Memory Space Analyzer V8.1 (Kanalyze)
http://download.microsoft.com/download/8/3/5/835efd1b-d801-4aa5-84ab-c4809fb493f4/Kanalyze%20First%20Step%20Guide%20V8.1.ppt

BitBlt
http://msdn.microsoft.com/en-us/library/ms532278(VS.85).aspx

DrvDitherColor
http://msdn.microsoft.com/en-us/library/ms793395.aspx

Debugging Deadlocks (No Ready Threads)
http://msdn.microsoft.com/en-us/library/cc267852.aspx

Error message when you try to print after you install the GDI security update 925902: "Stop 0x0000007F"
http://support.microsoft.com/kb/935843



Wayne's World of IT (WWoIT), Copyright 2008 Wayne Martin.

No comments:


All Posts

printQueue AD objects for 2003 ClusterVirtualCenter Physical to VirtualVirtual 2003 MSCS Cluster in ESX VI3
Finding duplicate DNS recordsCommand-line automation – Echo and macrosCommand-line automation – set
Command-line automation - errorlevels and ifCommand-line automation - find and findstrBuilding blocks of command-line automation - FOR
Useful PowerShell command-line operationsMSCS 2003 Cluster Virtual Server ComponentsServer-side process for simple file access
OpsMgr 2007 performance script - VMware datastores...Enumerating URLs in Internet ExplorerNTLM Trusts between 2003 and NT4
2003 Servers with Hibernation enabledReading Shortcuts with PowerShell and VBSModifying DLL Resources
Automatically mapping printersSimple string encryption with PowerShellUseful NTFS and security command-line operations
Useful Windows Printer command-line operationsUseful Windows MSCS Cluster command-line operation...Useful VMware ESX and VC command-line operations
Useful general command-line operationsUseful DNS, DHCP and WINS command-line operationsUseful Active Directory command-line operations
Useful command-linesCreating secedit templates with PowerShellFixing Permissions with NTFS intra-volume moves
Converting filetime with vbs and PowerShellDifference between bat and cmdReplica Domain for Authentication
Troubleshooting Windows PrintingRenaming a user account in ADOpsMgr 2007 Reports - Sorting, Filtering, Charting...
WMIC XSL CSV output formattingEnumerating File Server ResourcesWMIC Custom Alias and Format
AD site discoveryPassing Parameters between OpsMgr and SSRSAnalyzing Windows Kernel Dumps
Process list with command-line argumentsOpsMgr 2007 Customized Reporting - SQL QueriesPreventing accidental NTFS data moves
FSRM and NTFS Quotas in 2003 R2PowerShell Deleting NTFS Alternate Data StreamsNTFS links - reparse, symbolic, hard, junction
IE Warnings when files are executedPowerShell Low-level keyboard hookCross-forest authentication and GP processing
Deleting Invalid SMS 2003 Distribution PointsCross-forest authentication and site synchronizati...Determining AD attribute replication
AD Security vs Distribution GroupsTroubleshooting cross-forest trust secure channels...RIS cross-domain access
Large SMS Web Reports return Error 500Troubleshooting SMS 2003 MP and SLPRemotely determine physical memory
VMware SDK with PowershellSpinning Excel Pie ChartPoke-Info PowerShell script
Reading web content with PowerShellAutomated Cluster File Security and PurgingManaging printers at the command-line
File System Filters and minifiltersOpsMgr 2007 SSRS Reports using SQL 2005 XMLAccess Based Enumeration in 2003 and MSCS
Find VM snapshots in ESX/VCComparing MSCS/VMware/DFS File & PrintModifying Exchange mailbox permissions
Nested 'for /f' catch-allPowerShell FindFirstFileW bypassing MAX_PATHRunning PowerSell Scripts from ASP.Net
Binary <-> Hex String files with PowershellOpsMgr 2007 Current Performance InstancesImpersonating a user without passwords
Running a process in the secure winlogon desktopShadow an XP Terminal Services sessionFind where a user is logged on from
Active Directory _msdcs DNS zonesUnlocking XP/2003 without passwords2003 Cluster-enabled scheduled tasks
Purging aged files from the filesystemFinding customised ADM templates in ADDomain local security groups for cross-forest secu...
Account Management eventlog auditingVMware cluster/Virtual Center StatisticsRunning scheduled tasks as a non-administrator
Audit Windows 2003 print server usageActive Directory DiagnosticsViewing NTFS information with nfi and diskedit
Performance Tuning for 2003 File ServersChecking ESX/VC VMs for snapshotsShowing non-persistent devices in device manager
Implementing an MSCS 2003 server clusterFinding users on a subnetWMI filter for subnet filtered Group Policy
Testing DNS records for scavengingRefreshing Computer Account AD Group MembershipTesting Network Ports from Windows
Using Recovery Console with RISPAE Boot.ini Switch for DEP or 4GB+ memoryUsing 32-bit COM objects on x64 platforms
Active Directory Organizational Unit (OU) DesignTroubleshooting computer accounts in an Active Dir...260+ character MAX_PATH limitations in filenames
Create or modify a security template for NTFS perm...Find where a user is connecting from through WMISDDL syntax in secedit security templates

About Me

I’ve worked in IT for over 20 years, and I know just about enough to realise that I don’t know very much.