I'm running a Server Farm of 5 presentation layer machines
There are 55 different apppools, each is dedicated to a single .net applicatio
Only approximately 10 are running at any time and 8 of those get enough traffic to never shut down
1 in an unknown number of app pool restarts, from both timed recycle as well as timed shut down, the application is never able to answer a request
It appears to attempt the JITC and never completes it to give a response on the application
Have looked at this through the IIS discussion group, and they are pretty sure this is a code issue on our side, but since we have been unable to reproduce the problem in a development environment it is very difficult to troublshoot
The applications use the same basic code base with modifications in presentation
All uncached data comes from a Round Robined set of Middle tier servers, which switch machine after each request. Round Robin is provided by a list and custom code calls each machine in order
The only information we really have is that this occurs on recycles/restarts it does not matter if the server is being pounded, or is under light load
In looking at the output of IISState we found one of these errors for each thread that was being orphaned by IIS and showing up as Current Anonymous Users under Web Service Performance counters, as well as the Requests Current ASP.Net Counters
Thread ID: 1
System Thread ID: f0
Kernel Time: 0:0:0.
User Time: 0:0:0.
Other information: Thread is waiting for a lock to be released. Looking for lock owner
Owning thread System ID: 1f7
Thread Type: AS
Executing Page: ASP.dll symbols not found. Unable to locate ASP page
Continuing with other analysis
# ChildEBP RetAddr
00 009ffc3c 77f43741 SharedUserData!SystemCallStub+0x
01 009ffc40 77f5d64e ntdll!ZwWaitForSingleObject+0x
02 009ffc7c 77f42044 ntdll!RtlpWaitForCriticalSection+0x12
03 009ffc84 79e8d946 ntdll!RtlEnterCriticalSection+0x4
04 009ffccc 79e75739 aspnet_isapi!GetAppDomain+0x9
05 009fff1c 79e755e9 aspnet_isapi!HttpCompletion::ProcessRequestInManag edCode+0x13
06 009fff24 79e8cb80 aspnet_isapi!HttpCompletion::ProcessCompletion+0x2
07 009fff68 792e9155 aspnet_isapi!CorThreadPoolWorkitemCallback+0x2
08 009fff98 792e92cf mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0x2
09 009fffb8 77e4a990 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x12
0a 009fffec 00000000 kernel32!BaseThreadStart+0x3
When the site fails to restore, each new connection creates a Concurrent request that is never answered. We have allowed some of these to go past 3 hours by removing the machine they were running on from the Load Balance
The fun part is when this happens on a site that has been set to use a Web Garden, this will happen to just one of the threads, so the Automated checking software won't pick it up as a failure, because only every 3rd request is failing
Where can we go from here?
