Debugging a crashing IIS website
April 24, 2008 10:42 AM |
Comments (0) 
| Rate this article: 

Scenario

You have a web application that seems to be crashing from time to time.  You see events in the event log that demonstrate that the w3wp.exe process has crashed.  You have error reporting, but you don't see errors for the times that the w3wp.exe process is crashing.  The website is on a live server and you need to diagnose the issue without affecting the availability of the website.  If this is your scenario, then you will want to capture a crash dump and analyze the minidump file to isolate the problem.

Capturing crash dumps

The first thing you will need is the most recent IIS Diagnostics Toolkit.  This toolkit is designed to be installed on your web server.  This tool can be configured to capture IIS crashes.  It does this by hooking into the IIS process as a debugger.  Please note that this is not designed to be run in a standard production environment as having a debugger attached to your web process can introduce some instabilities.  But your website is crashing already, so attach the debugger and wait for a good crash dump, then detach the debugger while you analyze the dump.

IIS Diagnostics Toolkit (x86)
IIS Diagnostics Toolkit (x64)

Instructions:

  1. Download and install the tool on your web server.
  2. Run the Debug Diagnostics Tool.  The first time it is run, a wizard will be presented.
    1. Select Crash to set up a crash dump rule.  Click Next.
    2. Select "All IIS related processes".  Click Next.
    3. Select "Mini Userdump" for first chance exceptions.  Set a limit.  Click Next.
    4. Type in a rule name and path or use the defaults.  Click Next.
    5. Click Finish.
  3. Wait for an IIS crash.  Confirm the minidump file - it should have first chance exception or second chance exception in the filename.
  4. Click on the rule and select Deactivate after you have a good crash dump.  This will allow IIS to run in regular mode without having the debugger attached.

Note that if you have monitoring scripts or tools on the server that recycle the application pool or restart IIS when there are website problems, these tools might interfere with the crash dump.  When the ASP.NET worker process crashes, the debugger will be notified and it will dump the full process to a minidump file.  These files can be large, and it takes some time.  If your tools kill the process in the middle of generating the minidump file, you won't get the minidump file.

Workstation Setup

Once you get a good crash dump file, you will use your workstation to analyze it.  The tool you will be using is WinDbg, or the standard Windows Debugger.  You can download the appropriate release from the following locations:

Debugging Tools for Windows (x86)
Debugging Tools for Windows (x64)

Install the debugging tools for Windows on your workstation.  This will allow you to open the crash dump files.  Please note that this is a standard Windows debugging tool - it can debug all forms of processes.  Your ASP.NET application is written in .NET, and this debugger is not natively aware of that.  As such, the .NET framework comes with a debugging tool extension called SOS that can be used within Windbg to look into the process from a .Net standpoint.  This is likely going to be a bit confusing because you will use some native commands in Windbg as well as SOS extension commands.  Another confusing thing is that WinDbg will need the appropriate symbol files for the .net framework and other DLLs that are loaded into the IIS process.  Your workstation is not usually a server, and even then, symbol files are not usually installed on your workstation.  WinDbg has the capability of loading symbol files from a directory and the ability to download them from the web.  I personally prefer to download the full package on my workstation, but you can get WinDbg to load them on demand.

I prefer to set the symbol setting as an environment variable.  I set the following environment variable:
_NT_SYMBOL_PATH=SRV*C:\MyLocalSymbols*http://msdl.microsoft.com/download/symbols

You can also view Use the Microsoft Symbol Server to obtain debug symbol files

The symbols can also be downloaded as complete packages using the following url:
Download Windows Symbol Packages

Opening the crash dump file and enabling the SOS extensions

Using Windbg, select File | Open Crash Dump ...  Select your crash dump file and click Open.  This will load the crash dump file into WinDbg.  You should see some messages on your screen.  The top part of the screen is a message window and the bottom part of the screen includes a single command line interface. You enter commands in the bottom and observe the output in the top message window.  Anyways, you should see a message that states that there is an exception of interest in the file.  It will even instruct you to use the .ecxr command to retrieve the information for the exception.  You can try that if you wish.  However, unless you are an assembly expert,, this is not likely going to give you any useful information.  What you need to do is to load the SOS dll.  Enter .load SOS in the command window and hit enter.  You will see your command echoed in the message window but there is no output.

The native commands for WinDbg usually start with a period symbol.  SOS commands start with a bang, as in '!'.  You can enter the !help command to get a list of SOS commands and to confirm that the extension is active.

 

Getting Started with analyzing the crash

The most useful command that I usually start with is the Analyze command.  Enter !Analyze -v for a command to get a verbose analysis of the crash.

You should see the exception and the call stack.  This might be enough to help you understand the crash.

Other commands that are important are !do or !dumpobject.  You can !do any object by memory reference.  This allows you to dump the contents of any object.  This will help if you see some important memory addresses or exception details in the analysis.  This will use the SOS extension to look into the managed code and to examine .net objects that are in the memory heaps.

More information

I am not an expert on crash dump analysis.  I do it so seldomly that I really don't have the deep skills it takes to examine every type of problem.  Here are some related websites that have useful information on them related to these topics.

 

Links & Attachments 
Submit a Comment
Title:  
Name:    
Comment:    
Verification:

Type the characters you see in the picture below.