Help Debugging Memory Issues

[Rich Denis]

Hello,
I have been trying to solve a mysterious memory leak problem and was
hoping that you could help me out on my stuck point.
First a bit of background. We have two app servers in an app center
cluster that are each running a number of web applications. For some
reason one of the boxes' asp_wp process keeps recylcing on us. It seems
that the memory just grows and grows and then recycles on us.
I cam across and have read the Production Debugging for .NET Framework
Applications and found some very good information in trying figure out
where to start looking.

There was a discussion regarding looking at the difference between the
..NET # Bytes in all Heaps and the "Total Private Bytes" of the asp_wp
process. It said that if the total private bytes is growing faster than
the #Bytes in all heaps that it may be native code memory problem.

Current readings show the aspwp process in the task mgr as using 396
mb memory. The toal heap bytes shows only 72 Mb while the Private bytes
is reading 705 Mb. All numbers are stagnate at this point. My question
is:
How is it possible that the total private bytes add up to more than
what is being reported by the task manager. Is this something that I
should be looking at or should I really track memory growth from the
beginning of a clean reset.

Now also I am interested in using the Dump tool to catch when the
process recycles. If I run this script, what do I need to know if I
want to run this from a remote location. Can I use terminal services?
I seem to remember reading something about this but can not find it.
Also if I use the tool, what type of impact does it have on performance
when it is running waiting for the crash to occur?

One of the last questions I have regarding the allocation profiler is
similar to the one about the dump tool. If I have the tool running on a
production server what type of impact does it have on the performance of
the site. Will the performance be hurt so much that users will have an
adverse experience? I understand it is a necessary step, but I just
need to know how evil it is.

Thanks for any further elaboration on the great suite of tools and
documentation that you have created.

Rich

PS - is this the best group for this discussion?

[Felix Wang]

Hello Rich,

Thanks for posting. I would try to answer some of your questions.

- How is it possible that the total private bytes add up to more than what
is being reported by the task manager.

In Task Manager, are you looking at "Mem Usage" or "VM Size"? "Mem Usage"
is the physical memory the process is using. It should be "VM Size" that is
equivalent to "Private Bytes". We can add/remove columns by menu "View" |
"Select Columns..." in Task Manager.

- If I run this script, what do I need to know if I want to run this from a
remote location. Can I use terminal services?

You may refer to the help of WinDbg:

Crash Mode:

Because crash mode uses an "invasive" attach through the CDB debugger, it
does not work inside a Microsoft Windows NT 4.0 or Windows 2000 Terminal
Server session, as these operating systems will not permit a debugger in
one window station to attach to a process in a different window station.
Crash mode is supported in a Terminal Server session on Windows XP and
later versions of Windows. Hang mode will work inside a Terminal Server
session on any platform.

- What type of impact does it have on performance when it is running
waiting for the crash to occur?

I am not sure how to answer this. Of course there will be performance
penalty. However, it is quite difficult to quantify.

I hope the information will be useful to you.

Regards,

Felix Wang
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

[Rich Denis]

In article <Ot**************@cpmsftngxa10.phx.gbl>, v-
fe***@online.microsoft.com says...
Ok,
Thanks for the response. I have added the virtual memory size and now
the private bytes is in line with that. Regarding the private bytes,
here is what I now need to know.

The private bytes shows 1.1 Gb memory while the # Total Managed Bytes
only shows ~125 Mb memory. Why would there be such a big difference
between the managed and private? Is this normal? Are there any other
counters I can use to help figure out what happens.

Throughout the day the CPU will spike to 100% and the GC will start
running. It seems to hold all the resources up while the Memusage goes
down in size. The VM Size stays the same though.

What further can I look at. I plan on using the dump util early next
week but I hope to get more of a diagnosis now to see what it is that I
am looking at. Thanks.

Rich

Hello Rich,

Thanks for posting. I would try to answer some of your questions.

- How is it possible that the total private bytes add up to more than what
is being reported by the task manager.

In Task Manager, are you looking at "Mem Usage" or "VM Size"? "Mem Usage"
is the physical memory the process is using. It should be "VM Size" that is
equivalent to "Private Bytes". We can add/remove columns by menu "View" |
"Select Columns..." in Task Manager.

- If I run this script, what do I need to know if I want to run this from a
remote location. Can I use terminal services?

You may refer to the help of WinDbg:

Crash Mode:

Because crash mode uses an "invasive" attach through the CDB debugger, it
does not work inside a Microsoft Windows NT 4.0 or Windows 2000 Terminal
Server session, as these operating systems will not permit a debugger in
one window station to attach to a process in a different window station.
Crash mode is supported in a Terminal Server session on Windows XP and
later versions of Windows. Hang mode will work inside a Terminal Server
session on any platform.

- What type of impact does it have on performance when it is running
waiting for the crash to occur?

I am not sure how to answer this. Of course there will be performance
penalty. However, it is quite difficult to quantify.

I hope the information will be useful to you.

Regards,

Felix Wang
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

[Felix Wang]

Hi Rich,

Thanks for your update. The high virtual memory usage should be abnormal.

On the other hand, we should look at the data based on a time series,
instead of a single point. If there is an unmanaged memory leak problem
(which looks like), I would suggest that we check unmanaged resources in
your web application, e.g. COM components, PInvoke usage and so on.

As far as I know, it is not convenient to track unmanaged memory leak
issues with dumping. We have a tool named "LeakDiag". However, it seems to
me that the tool is not available publicly. You may contact Microsoft
Product Support Services.

Regards,

Felix Wang
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

[Rich Denis]

Felix,
Thanks for the response. I have some other information that may be
you can help me make sense of. Here it goes:

As I said before we have this code running on multiple servers in a
cluster (2 to be exact). Neither of the servers is the same when it
comes to processor size or memory. Here are the specs for reference:
Box A:
866mhz 2G ram
Box B:
1.1 ghz 3G ram

Ok so on Box A is the box that has the high virtual memory usage on it.
When looking at the task mgr there is definitely more vm allocated than
mem usage. On every other box that I have looked at (box b inculded)
when looking at the same metrics, the VM is NEVER higher than the mem
usage. The way that I understand VM usage it is the total amount of
memory that the process is using, both in RAM and on Disk. If that is
the case then why would the VM number be lower than the physical memory.
Is it possible for the process to hold onto some physical memory that is
not counted when doing VM counts? This is what it would seem to me but
I just wanted clarification.

As to the potential issue with virtual memory that was not be freed, I
have narrowed it down to what I think is bad coding when it comes to
closing sql connections. When looking at the total # pooled connections
on the offending server, when it is under high memory utilization, there
are roughly 1100 pooled connections. Under higher load that number
keeps increasing. Under moderate load that number stays around for a
while and will scale back some under no load times. I have found at
least one page in an app that seemed to be opening 5-9 connections per
page invocation and not releasing any of them (ie the connection would
fall out of scope without call or dispose being explicityly called.)

Back to the counter that showed a high number of connections opened, why
is it that the number does not go down after an IISReset? How could I
look further into what is happening in the connection pool? I would
have expected that after a reset that the number would dwindle back down
to the zero area but this did not happen even though the asp_wp memory
reset itsself.

Lastly, with me looking back at a connection problem, is this something
that can be diagnosed with a dump or should I still contact PSS to get
me a lead on that LeakDiag?

Thanks again for any help,

Rich

In article <Nf**************@cpmsftngxa10.phx.gbl>, v-
fe***@online.microsoft.com says...

Hi Rich,

Thanks for your update. The high virtual memory usage should be abnormal.

On the other hand, we should look at the data based on a time series,
instead of a single point. If there is an unmanaged memory leak problem
(which looks like), I would suggest that we check unmanaged resources in
your web application, e.g. COM components, PInvoke usage and so on.

As far as I know, it is not convenient to track unmanaged memory leak
issues with dumping. We have a tool named "LeakDiag". However, it seems to
me that the tool is not available publicly. You may contact Microsoft
Product Support Services.

Regards,

Felix Wang
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

[Felix Wang]

Hi Rich,

Thanks for your update.

In fact, the term used in Task Manager is inaccurate. In Task Manager, "VM
Size" in fact should be "Private Bytes". "Mem Usage" should be "Working
Set". You may refer to the "Explain" button in "Add Counters" dialog of
Performance Monitor for detailed definition of the terms.

As far as I know, if we run IISReset, the worker process of ASP.Net should
be terminated as well. All the resources should be free, since the process
is shut down.

Regarding the connections, have you explicitly called
"SqlConnection.Close()" method to close the connections? Will it help if we
do so?

Regards,

Felix Wang
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

[Rich Denis]

Felix,
Sorry it took so long to get back to you. Life got in the way. Ok so
here is where we are today. The memory on one box seems to grow without
ever being reclaimed before the process is recycled. I had tried to get
a dump of the aspnet_wp.exe process by running in adplus in hang mode.

Oddly, when I created the dump while the memory VM usage was around
1.1 Gb the dump that was generated did not create the dmp file. I had
to reset the process and let the memory build up a bit in order to get a
dump. At lower memory utilizations, the .dmp file was created
correctly. Once the memory reached around 400Mb I was not able to get a
*.dmp file to generate.

**1. Are there any known issues where .dmp files are not created? The
folder structure gets created and the log file even shows up but not dmp
file.

**2. Is there anything that the log may tell me that is useful or do I
need the .dmp file in question.

Regarding the reading of the dump file, is there anything that I need
to do other than what is outlined in the "Production Debugging for .NET
Framework Applications"?

Thanks.

Rich
In article <KW**************@cpmsftngxa10.phx.gbl>, v-
fe***@online.microsoft.com says...

Hi Rich,

Thanks for your update.

In fact, the term used in Task Manager is inaccurate. In Task Manager, "VM
Size" in fact should be "Private Bytes". "Mem Usage" should be "Working
Set". You may refer to the "Explain" button in "Add Counters" dialog of
Performance Monitor for detailed definition of the terms.

As far as I know, if we run IISReset, the worker process of ASP.Net should
be terminated as well. All the resources should be free, since the process
is shut down.

Regarding the connections, have you explicitly called
"SqlConnection.Close()" method to close the connections? Will it help if we
do so?

Regards,

Felix Wang
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

[Felix Wang]

Hello Rich,

Thanks for your reply. I am not aware of any known issues about large dump
files. Do we have enough disk spaces? When the dump is generated, we should
observer disk activities.

The log file may not provide enough information regarding the problem.

I would still recommend that you contact Product Support Service for the
memory leak issue.

Regards,

Felix Wang
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.