IIS Errors

[Seneca]

We have a webspeed application that is crashing frequently. It is returning errors to the user in the browser (I believe it is just page can not be displayed 404 errors), and as putting many of the following errors in the event log:

====================================================
Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1011
Date: 11/1/2006
Time: 4:36:47 PM
User: N/A
Computer: TS2
Description:
A process serving application pool 'DefaultAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '3256'. The data field contains the error number.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 6d 00 07 80 m..€
=====================================================


Eventually after a certain number of these errors, it will shut down the W3SVC and give the user a page that says "Service Unavailable" in the browser, and put this error in the system log:

=====================================================
Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1002
Date: 11/1/2006
Time: 4:59:41 PM
User: N/A
Computer: TS2
Description:
Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

=====================================================

We are running Progress/Webspeed 9.1D, with 25 agents. Webspeed (DBs, Trans Svr, Name Server, Broker, etc.) is local to this server with the exception of a DB on a Unix Box (Solaris 2.6) Running Progress 8.3E. Server is Windows 2003 Server Enterprise Edition - IIS 6.0.


Any ideas what the underlying cause of this might be, or how to correct it? I have looked far and wide for information regarding the event id's above, and can't really seem to find anything useful.

Any input would be appreciated. Thanks.

 

[Casper]

What does your webspeed broker and server log say?
And how do you connect with webspeed to a 8.3E database?

Regards,

Casper.

 

[Casper]

From a quick google:

This error, eventId 1011, is a generic error thrown by the world wide
web publishing service when the IIS6 Application pool (w3wp.exe)
crashes when an http request is being processed. Without mattering
what the cause of the crash is, this error is logged. If you just
cancel the w3wp.exe process from the task manager when a request is
being process, this error will be logged.

(source= http://eggheadcafe.com/ng/microsoft.public.inetserver.iis/post203164.asp
)


Regards,

Casper

 

[Casper]

Did you already had a look at: http://support.microsoft.com/kb/885654/en-us

Regards,

Casper.

 

[Seneca]

Just using a progress connect statement to connect to the 8.3E DB.
There aren't any errors in the server/broker logs that seem to directly pertain to the errors in the system logs. We have gotten some errors for years now that look like this:

S-0009>(Nov 6, 2006 12:06:35:419) [1916] WTA: WebWrite() failed during npp_send with error: -70 size: 1739 sentbytes 0 retrycount 1. (6404)

along with an occassional

S-0015>(Nov 6, 2006 12:04:55:660) [5992] WTA: npp_send() failed while sending terminate message! Error: -70. (6397)

We have had problems with all agents going busy for years now, but it was for the most part not real frequent.

I did look at the microsoft KB, but it doesn't seem as if it would apply, this machine is not a domain controller.

 

[Casper]

What is the state of the agents when these errors happen? Are they all busy?
What about the response times of the agents? (avg. request time, min, max)
What changed when these started to appear? (IIS 5 --> IIS 6?).

What service pack do you have with 9.1D?

Casper

 

[Seneca]

No, when these errors occur the agents are not all busy. Often there are only 1 or 2 that are busy at the time.

Average request/response times - not sure - is there somewhere you can see this?

No specific change was made that I can correlate to this new problem occuring.

I forget how to see the patch level on progress - would it be in the config file? That just shows 9.1D.

 

[Casper]

If you do a wtbman -i servicename -q then you can see the request times.
For progress version and patch level look in $DLC/version.

Progress 9.1D06 and higher is certified for Windows TS 2003.

I asked those questions to see if it is a Microsoft or a Progress issue.

Anything in the http.sys log which says some more about the 1011 error?

Regards,

Casper.

 

[Seneca]

Thanks for the replies, I appreciate your help!

Progress is version 9.1D07

The httperr#.logs have errors that correspond, here are a couple samples:

2006-11-01 19:30:39 72.43.26.210 59741 172.16.4.48 443 HTTP/1.1 GET /scripts/wsisa.dll/WService=pt/wui/kittype_selection.w?web_uid=010204EWK&web_suid=jKdJazjjRbUhXnkz& - 1 Connection_Abandoned_By_AppPool

2006-11-01 19:30:39 72.43.26.210 58325 172.16.4.48 443 HTTP/1.1 GET /scripts/wsisa.dll/WService=pt/dst/part-search-b.w?web_uid=010204EWK&web_suid=jKdJazjjRbUhXnkz& - 1 Connection_Abandoned_By_AppPool

2006-11-01 19:30:39 72.43.26.210 59187 172.16.4.48 443 HTTP/1.1 GET /scripts/wsisa.dll/WService=pt/dst/part-search-b.w?web_uid=010204EWK&web_suid=jKdJazjjRbUhXnkz&sort_by=price&view_results=Search%2FView+Results&col_1_subcat=Switch&col_2_one=3com&match_item=&match_descrip= - 1 Connection_Abandoned_By_AppPool


output from wtbman looks like this:

Broker Name : ccm-live
Operating Mode : Stateless
Broker Status : ACTIVE
Broker Port : 29365
Broker PID : 3856
Active Agents : 25
Busy Agents : 7
Locked Agents : 0
Available Agents : 18
Active Clients (now, peak) : (2, 5)
Client Queue Depth (cur, max) : (0, 0)
Total Requests : 25021
Rq Wait (max, avg) : (16 ms, 0 ms)
Rq Duration (max, avg) : (18 ms, 0 ms)


I had once heard there is a memory leak with wsisa.dll - not sure what version that applied to. I was considering switching to cgiip.exe to see if that helps - any thoughts on that?

 

[Casper]

cgiip is more relliable with heavy load of data.

There are some issues with wsisa regarding to message Size in IIS.
from KB 20951:

There is a limit on the size of data it can read in each message. This only happens with the WSISA.DLL and causes the messenger
to lock up and the session to hang.

Maybe it's better to use cgiip which is more reliable then wsisa but I don't know if this solves your problem.

We use cgiip and apache on Linux, so I'm not an expert in this field BTW

The error: Connection_Abandoned_By_AppPool means that there was a crash in the user code running in the apppool. Most likely this error is caused by your webspeed application and not by IIS.

To find the real reason of these kind of errors you have to trap the next crash and debug the stacktrace. You can use debugtools like debugdiag: http://www.microsoft.com/downloads/...bc-376b-4a54-95aa-73c9156706e7&displaylang=en
or IIS state: http://www.iisfaq.com/default.aspx?view=P197

to create the stacktrace log. Hopefully you have some guy who can do this.

The code which produces these errors (part-search-b.w and kittype_selection.w) usually runs fine?

Can you reproduce the error at will?

I'm afraid I can't help you with analyzing stack trace logs.

HTH (a bit),

regards,

Casper

 

[Seneca]

Yes, they usually run fine. Those were just 2 samples. The errors are generated all over the place, not just those 2 - there are probably a dozen if not more procedures referenced in the httperr log. The biggest thing that has changed overall is the load on the system in general. It is supporting more users than ever.

I cannot reproduce at will, no. It is frequent, but still sporadic.

I'll take a look at the debug/stacktrace stuff and see if anyone can make sense of it.

Sadly, the first thing I think we are going to try is to throw some hardware at this server - upgrade the RAM and see what that does. I think if that has no impact, we may go to cgiip.exe and see what that gives us...

Thanks for your input - I appreciate it!