1
We are seeing some kind of resource leak in our performance lab running an ASP.NET (2.0) application that sends and receives messages from 2 public MSMQ queues. Here's a brief summary of what are application does:
1. user provides input on a form on an aspx page and clicks continues
2. on postback, the page creates a c# .net object and calls a method on it to process the data provided by the user
3. the object in turns creates an MessageQueue object and uses it to send a message to an MSMQ queue. It then creates another MessageQueue object to receive a response message from a different queue. our object does not use events or callbacks. It calls ReceiveByCorrelationId to get its message back.
4. our code calls close and dispose on all queues and messages as soon as they are not needed anymore.
5. because of legacy code and need for backwards compatibility, we use the activeX formatter for the outgoing and incoming MSMQ message.
about our setup:
* our app runs on a win2k advanced server.
* the queues are on a separate server also running win2k
* the DC is yet on another server also running win2k
* all servers are on the same LAN and run MSMQ 2.0
When we run our application in our performance lab using a stress test tool, we are seeing the following behavior:
1. everything runs fine and fast until about 30 to 40 minutes into the test (9 minutes into the test, all test users are fully in the test executing all steps of the test script ). Then we see a huge slow down in response times and after a while aspnet_exe restarts. from that point on, the app never truly recovers and runs slow.
2. we have run TMQ STATE at various times during the test and see a constant increase in the number of handles and threads during the test. For example, TMQ state reports that at 15 minutes into the test, the number of threads is 49 and the number of handles is 383. At 41 minutes into the test the number of threads is 68 and the number of handles is 560.
3. we run only 40 test users during the test so at most there are only 40 concurrent calls happening to MSMQ.
4. The Pool Paged Bytes memory perf counter is creeping up during the test but only from 40 MB to 45 MB or so.
5. There are no MSMQ errors in the event log
6. after the problem occurs, our app logs a number of MSMQ timeout errors indicating that it did not get a response message within a given amount of time (set at 100 seconds for this test)
7. if we run a similar script that does not make calls to MSMQ, we do not see this problem.
8. we turned MSMQ logging on and see many errors/events being recorded in the log during the test
These numbers seems to indicate to me that we are (1) leaking some resource related to MSMQ and (2) that we hit a threshold for this resource 30 to 35 minutes into the test.
We have captured a bunch of other counters and data that we can share if this would help. Note that we have read all MSDN and forum posts related to MSMQ and resources, but none are helping (our messages our way less than 4 MB in size, we don't use callbacks, we don't get a "insufficient resources" error, etc.)
Does anyone have experienced similar issue? Also, does anyone know how to interpret TMQ State data (what are the "number of threads" and "number of handles" data for example) and MSMQ logs?
any help would be greatly appreciated.
~
