Iis logs 500 error

If your Internet Information Services (IIS) produces a 500 – Internal server error, your website is in serious trouble. Debugging an IIS 500 – Internal server error can take some time, so you’d better be prepared for the worst-case scenario. You don’t want to research how to deal with this error under time pressure.

In my previous posts, you learned about detailed errors and failed request tracing in IIS (Internet Information Server). I recommend reading those articles first before you proceed with this one.

Cause of 500 – Internal server error

This is the most common error you will encounter with any website hosted with IIS. In most cases, a developer messed up. Thus, the fastest way is often to simply reverse the last action taken, such as restoring an earlier version of your web application. Once your system is running again, you can investigate the cause of the error on your test side in peace.

The HTTP 500 error is a server-side error. While we understand that the problem is on the server end, the error is usually ambiguous. It doesn’t exactly tell the administrator what is wrong with the server. Thus, debugging a 500 – Internal server error often takes some time.

Debugging an IIS 500 – Internal server error

Since the above error doesn’t really tell what’s actually wrong with the server, we need to enable detailed errors, as discussed in my previous post. Once detailed errors are enabled, you will see more detailed error information, including an HTTP substatus code. Sometimes even the detailed errors don’t show any useful information right away. For example, see the following screenshot:

Here I am getting: The page cannot be displayed because an internal server error has occurred. There is no HTTP status code or substatus code listed on the error page. If you get such an error even when detailed errors are enabled, right-click anywhere in the browser window and select Inspect (or press F12).

This opens the developer tools in your browser window. Now, click the Console tab. The actual error thrown by the web server is displayed.

To further understand the exact cause of 500 errors, enable Failed Request Tracing, as discussed in my previous post. Now, try to replicate the problem. If you can replicate it, open the newly generated XML log file in a web browser. The following screenshot shows the actual cause of a 500 – internal server error with a substatus code of 19 (HTTP 500.19 error):

Usually, substatus code 19 indicates that the configuration data is invalid. This could be due to some malformed or unidentified element in a server-level config file (ApplicationHost.config) or website-level config file (web.config). If you take a closer look at the ConfigExceptionInfo field of the log file, you will find the exact line number (6 in our case) in the web.config file that caused the exception. Now let’s take a look at the web.config file itself.

Here, you can see that the developer tried to add a mime type in the config file, but it was already defined in the server-level configuration file (i.e., ApplicationHost.config). Therefore, the Cannot add duplicate collection entry of type ‘mimeMap’ with unique key attribute ‘fileExtension’ set to ‘.mp4’ exception was returned. Furthermore, if there is some unidentified element, a syntax error, or even a typo in the web.config file, you will most likely get a similar error.

Resolving an IIS 500 – Internal server error

To resolve an IIS 500 – Internal server error, you could simply remove the line that is causing the exception. Alternatively, if you don’t want to remove this line for some reason, add the following code right above line 6 in web.config:

<remove fileExtension=".mp4" />

By doing this, you are essentially overriding the server-level element. In the end, your web.config file should look as shown below:

Now refresh the page, and the error should go away. This was just one example of resolving a 500.19 error. If you get a 500 error with a different substatus code, use the same approach to troubleshoot the problem.

Common 500.x substatus codes

The following table covers some of the most common HTTP 500 substatus codes, along with their probable causes and troubleshooting advice:

Subscribe to 4sysops newsletter!

Once you troubleshoot the problem, don’t forget to disable Failed Request Tracing and revert the detailed errors to custom errors on your web server.

Over the years, Microsoft IIS has grown to be an exceptionally flexible product, supporting numerous technologies, products, extensions, and configuration options. IIS can do almost anything, from servicing simple static pages to being a fully blown, enterprise-grade reverse proxy server. Like any complex product, though, IIS requires troubleshooting on occasion. Here we’ll look at some of the most common challenges that web developers and administrators face when deploying and developing applications for IIS and how to troubleshoot them properly.

Runtime Errors and Exceptions

The most common issues customers encounter when deploying or developing for IIS are runtime errors and exceptions, which include various errors from the 40x and 50x families of HTTP errors. These errors include the common 404 and less-common 400 errors, as well as the dreaded 500 internal server error.

The appearance of these errors varies depending on the IIS server’s configuration. The HTTP 500 error, which is typically an application or pipeline error where something failed before the application sent any response, could manifest itself as a simple error in the browser, such as the examples shown in Figure 1.

However, if the error occurs in a managed code .NET application, the error might look like that shown in Figure 2.

The first step in figuring out such errors is a differential diagnosis. This means that we need to deduce from user reports that the site hangs are actually IIS worker process (w3wp.exe) crashes (which are what happens when a web application experiences a major error) or possibly something else, such as a configuration problem of some kind. We can find this information from multiple sources:

• IIS logs
• by disabling custom errors to display detailed error pages
• failed request tracing
• Windows Application Event Logs

Examining the IIS logs is a good first step, as the logs are easy to read and would confirm whether or not the error is really a 500 error. If it is a 500 error, the log line would show the HTTP status code to be 500 0 0, as in Figure 3, instead of the expected 200 code.

The next step is to look for more details about the error. It’s not uncommon for administrators who see this error to assume that it portends the worst. The error seems to suggest there’s a problem with the code, and many system administrators are not trained to read or debug code. In reality, this type of error can be really simple to understand once you realize that the seemingly meaningless XML tags are not the actual error but rather a message telling you that the default configuration prevents IIS from actually showing you what’s wrong. (Why so? Because we don’t want strangers who visit the site to get details about your code!)

To see what’s really going on, we need to configure the server to be more informative. This is done by adding a section to the site’s web.config file. The steps are:

1. Open the site’s web.config file with an editor of your choice.
2. Inside the system.web section of the file, add the line . For example:

Note: The reason we use RemoteOnly here is to display the error only on your local server and not to users trying to visit the site. If you want the error to be displayed publicly, or if you are setting the configuration on a Windows Azure Web Sites server (to which you don’t have direct access), you would set the mode to Off.

Now, if you connect to the site again, the error page will show more information, which could potentially solve the riddle right then and there. For example, the error page shown in Figure 4 indicates that the code is trying to populate a variable that wasn’t declared.

An alternative way to view this error, if you prefer, is to inspect the Application event log on the system. When an application error such as the one shown in Figure 4 occurs, it will be logged as coming from ASP.NET. In such a situation, the event details would contain the same information, as shown in Figure 5. (The code filename, line, and error details are highlighted in yellow.)

Another common variation of an internal server error is error number 500.19. This error is also one you might run into when you’re using Azure Web Sites and developing advanced applications. The error looks like the page shown in Figure 6.

Error 500.19 is actually an entire family of errors pertaining to problems in the website’s configuration that’s specified in web.config. Figure 6 shows a permission problem, which could happen if the access control over the website’s folder is not set correctly and IIS cannot read the configuration.

Another variation of this is error 0x80070021, which happens when a section you are trying to configure in a site’s web.config conflicts with a setting set in the higher-level ApplicationHost.config file. For example, on an IIS server, the MIME types for content compression are set by default to disallow a site-level web.config configuration. If you try to add a configuration section to web.config without adjusting the inheritance settings first, you would receive this error.

Yet another variation of error 500.19 is 0x80070032, which happens when a section is missing configuration information. It’s not always easy to figure out what you did wrong, especially if your configuration is complex or if you’re trying to configure something that doesn’t have a lot of examples in the documentation, so for this error, an incremental approach is typically useful.

Although it would be impossible to cover every conceivable variation of the 500 error, the key lesson here is to get IIS to show you the complete details. Once you have those, the accompanying text often will provide all the information you need. If you need more information, you can search for the error code or keywords online and in that way find enough information to pinpoint the actual problem. For other errors, the error codes are usually more direct—see this Microsoft article for a list of the most common codes and additional information about them. Also see this list of Win32 errors, which can also help you learn more about many issues.

Crash and Hang Problems

A process crash happens when there is an error in a web application, which causes an exception, leading to termination of the IIS worker process (w3wp.exe). Unlike a regular application crash, where the user can easily see the application disappearing, with a web-based app there’s no such disappearance visible to the naked eye, so things can sometimes get murky. Sometimes, the user might get a Service Unavailable message in the browser (which might be totally unclear and confusing to an average user, but at least it’s something to start with). Other times, IIS will start a new w3wp.exe process to service the request. Although the second option seems desirable, as the user doesn’t run into errors, in reality, it could be almost as bad as the Service Unavailable error, because unbeknownst to the administrator or application developer, the application might be crashing all the time, providing a slow and unreliable service.

The good news is that the server will log events even if the user doesn’t experience a problem, which helps the administrator or developer catch such errors early. Typically crash-related events would be event ID 5009, which indicate a worker-process crash, or event IDs 5010 and 5011, which indicate that the process-activation service could not get a reply from a worker process. When any of these errors occur, they are often accompanied by an event ID 1000, which is for an application crash.

Having a process crash occasionally doesn’t always warrant a full investigation. No code is perfect, and if you run into a crash a few times a month, it’s usually not the end of the world. IIS recovers and starts a new w3wp.exe process upon the next request that comes in for the application. Naturally, though, if the crash causes data loss, or if it’s frequent enough to cause turmoil within your user base, then the investigation typically involves getting a crash dump, which contains a snapshot of the process when the exception occurs.

Examining a crash dump can be a challenge, as the connection between what’s in the dump and the root cause isn’t always evident. If you are the application developer, an alternative to this is to write your application code to include trace information that might make it possible to troubleshoot directly, or at least help in the process. This would involve adding to each function in your code a call to log actions and key variables into a text file (typically through a dedicated logging function). For example, when you run a loop, you can have the function log a line for each iteration, like this:

fTraceLog("Starting going through aRequestItems array at " & now)
For I = 0 to ubound(aRequestItems)
     fTraceLog("Request Item " & I & " contains " & aRequestItems(i))
     sRequestAction=http:// & cstr(aRequestItems(i)) & "/SetRecords.asp" & iJIDNumber
next
fTraceLog("Finished going through aRequestItems array at " & now)

With this code, if there is a problem with the loop, the last item in the trace could potentially reveal what it is. Keep in mind, though, that this sort of tracing could lead to a high strain on the server, so it should normally be turned off until an investigation is required.

The tricky part about crash dumps is that they need analysis, which is traditionally seen as a developer-level task. Fortunately there are some advanced tools that make crash-dump analysis more accessible for non-developers. The free Debug Diagnostic Tool (DebugDiag) contains some pretty powerful automatic analysis tools, which can shed light on a lot of issues without requiring you to look at a single line of code.

To use DebugDiag, you need to create a rule, which will instruct the debugger to detect when IIS crashes and collect the crash dump. To do so, follow these steps:

1. Launch DebugDiag from the Start menu or Start screen.
2. Click Crash, and then click Next.
3. Select the desired Target Type, then click Next.
4. In the Advanced Configuration (Optional) dialog box, click Next.
5. In the Select Dump Location And Rule Name (Optional) dialog box, click Next.
6. Click to select the Activate the rule now check box, then click Finish.

After you have set up the rule, you need to reproduce the issue, if you can, or wait for IIS to crash again. Once that happens, the rule will cause a creation of a dump, which you can now analyze. To perform the analysis, follow these steps:

1. Run the DebugDiag 2.0 analysis tool from your Start menu or Start screen. You’ll see the DebugDiag Analysis screen, as shown in Figure 7.
2. Click Add Data Files to tell the tool which file to analyze.
3. Select which Analysis Rules you want to run (we are looking into a crash, so you would typically check the Crash/Hang Analyzers rule).
4. Click Start Analysis to analyze the dumps.

Once the analysis is finished, DebugDiag generates an HTML report and opens it in your browser automatically. The report lists all the problems found and also provides suggestions for solving them. Based on the problem’s severity, the issues found are categorized in three types: Error, Warning, and Information.

Depending on the issue, the report could be a simple read, but it could also leave you more confused about the problem. As with any other type of troubleshooting, you’ll need to carefully read the data in the onscreen report to gain a better understanding of the problem. Used properly, search engines can also be helpful. For example, you might see this message displayed in the report:

The application domain is restarting because of reason:- A change was made to the Bin folder or to files in it

Lost session variables and appdomain recycles for more details on this issue.

As you can see, this message suggests what the problem is and even provides a link to further reading. Here’s another useful message:

The requests start getting queued if all the threads available to process the request are busy. Look at the callstacks of the threads which are processing the request to see why they are running for a long time. Also, look at the ASP.NET Request Queue Report to see the ASP.NET Threadpool settings and the runtime for which the requests are getting queued.

There’s More to Learn

Naturally, we cannot cover the entire range of possible issues with IIS here. However, as you use DebugDiag more, you will find that it’s exceptionally intuitive and helpful in IIS debugging. For additional information about debugging IIS issues, also see the following articles from Microsoft’s knowledgebase can guide you further. These articles also discuss scenarios other than a simple crash:

• How to use the Debug Diagnostics tool to troubleshoot an IIS process that stops unexpectedly
• How to use the Debug Diagnostics tool to troubleshoot a process that has stopped responding in IIS
• How to use the Debug Diagnostics Tool to troubleshoot high CPU usage by a process in IIS
• How to use the IIS Debug Diagnostics tool to troubleshoot a memory leak in an IIS process

About the Authors

Erez Benari is a senior web technology professional with more than 20 years of experience. During his career, Benari worked for some of the largest companies in the world and has an extensive background in business, economics, communications, and education. He is a resident of Washington, where he works for Microsoft as part of Windows Azure. In his role, Erez is a program manager for IIS, Microsoft’s web server platform, as well as Windows Azure Web Sites. Benari is also a journalist, a published author, and a professional stand-up comedian.

Richard Marr is a senior Escalation engineer with Microsoft support. He has worked for Microsoft since 2000. During his career, he has supported Microsoft Access, IIS, and ASP.NET and now works with Windows Azure Web Sites. Richard is a graduate of Mount Saint Vincent University in Canada and holds a Bachelor of Business Administration. He lives with his family in Texas.

Wei Zhao is a Shanghai-based Escalation Engineer for Microsoft. In his role, he helps Microsoft’s customers design, develop, deploy, and operate enterprise solutions using IIS, ASP.NET, and Windows Communication Foundation (WCF). Most recently, Zhao is also focusing on Windows Azure Web Sites.

If you’ve  ever had to troubleshoot issues with IIS you’ll known that you are often drawn to looking at IIS Logs.    This post is about a script that I modified to search IIS logs for specific errors.

To begin with I found this article on StackOverFlow that got me started on this path.

To begin with we need to get the webadministration module so we can get the name of the website we want to get the log file for:

Import-Module WebAdministration

$site = Get-Item IIS:Sites$website
$id = $site.id
$logdir = "$($site.logfile.directory)w3svc$id"

Now that we have the  The $logdir (log directory) we can now put the rest of the file name together by getting the date:

$File = "$logdiru_ex$(((get-date).adddays(-$days)).ToString("yyMMdd")).log"
s

Assuming you are using a logfile per day the name of the log file is:

u_ex(yyMMdd)  which should be something similar to this: u_ex170824.log

Now that we have our log file we need to strip off unnecessary lines.  Specifically the First three lines that start with #S, #D, or #V which ends up being the Sofware, Version and Date items at the top of the log.

Then we’ll need to Build the columns based on the #fields value in the Log. That way each field in the results can be arranged into columns so we can sort our data based on what the item in the field is.

The additional column we’ll add is the name of the log file.

$Log = Get-Content $File | where {$_ -notLike "#[D,S-V]*" }

$Columns = (($Log[0].TrimEnd()) -replace "#Fields: ", "" -replace "-","" -replace "(","" -replace ")","").Split(" ")

$Columns += 'LogFile'

$Count = $Columns.Length

Now that we have the column titles now it’s time to filter out the log and only get the lines that have the value we want.  In my case I was searching for the Error code of 500.

$Rows = $Log | where {$_ -like "*$errorType 0 0*"}

Now that we have all the data we want in the row’s variable we can now construct a table of Columns and rows.

$IISLog = New-Object System.Data.DataTable "IISLog"

foreach ($Column in $Columns) {
$NewColumn = New-Object System.Data.DataColumn $Column, ([string])
$IISLog.Columns.Add($NewColumn)
}
# Loop Through each Row and add the Rows.
foreach ($Row in $Rows) {
$Row = $Row.Split(" ")
$AddRow = $IISLog.newrow()
for($i=0;$i -lt $Count; $i++) {
$ColumnName = $Columns[$i]
if($ColumnName -eq 'LogFile')
{$AddRow.$ColumnName = $file }
else {$AddRow.$ColumnName = $Row[$i]}
}
$IISLog.Rows.Add($AddRow)
}

$IISLog

Full function is below:

  function get-ErrorLogs
  {
  param($website = 'myWebSite', $errorType = '500',[int] $days =0)

    Import-Module WebAdministration

    $site = Get-Item IIS:Sites$website
    $id = $site.id
    $logdir = "$($site.logfile.directory)w3svc$id"

    $File = "$logdiru_ex$(((get-date).adddays(-$days)).ToString("yyMMdd")).log"
    $Log = Get-Content $File | where {$_ -notLike "#[D,S-V]*" }
    $Columns = (($Log[0].TrimEnd()) -replace "#Fields: ", "" -replace "-","" -replace "(","" -replace ")","").Split(" ")
    $Columns += 'LogFile'
    $Count = $Columns.Length
    $Rows = $Log | where {$_ -like "*$errorType 0 0*"}
    $IISLog = New-Object System.Data.DataTable "IISLog"
    foreach ($Column in $Columns) {
      $NewColumn = New-Object System.Data.DataColumn $Column, ([string])
      $IISLog.Columns.Add($NewColumn)
    }
    foreach ($Row in $Rows) {
      $Row = $Row.Split(" ")
      $AddRow = $IISLog.newrow()
      for($i=0;$i -lt $Count; $i++) {
        $ColumnName = $Columns[$i]
        if($ColumnName -eq 'LogFile')
        {$AddRow.$ColumnName = $file }
        else {$AddRow.$ColumnName = $Row[$i]}
      }
      $IISLog.Rows.Add($AddRow)
      }

    $IISLog
  }

Now that I have this in a full function I can just call it like this:

get-ErrorLogs -website “Website2” -errorType 500 -days 4

And get results similar to this:

I hope this helps someone
Until then keep Scripting
Thom