Why is OutOfMemoryExce ption thrown?

[Lamont Sanford]

I can't figure out why once or twice per week, my C# server application
throws OutOfMemoryExce ptions. There is plenty of memory in the machine. As
far as I can tell, when the exception is thrown, only 400 megs of 1gig total
is in use.

The exception is thrown while trying to update List<>. Here's a bit of the
stack trace:

===
System.OutOfMem oryException: Exception of type 'System.OutOfMe moryException'
was thrown.
at System.Collecti ons.Generic.Lis t`1.set_Capacit y(Int32 value)
at System.Collecti ons.Generic.Lis t`1.EnsureCapac ity(Int32 min)
at System.Collecti ons.Generic.Lis t`1.Add(T item)
at Mosaic.Core.Sta tUtils.ExtractR ORs(MiniPerfIte m[] perfItems, Int32
startIdx, Int32 endIdx, Boolean useNAOnNonPosBa l) in
C:\Development\ Mosaic\Mosaic-AM\Mosaic.Core. Shared\Code\Uti lities\Statisti cal\StatUtils.c s:line
38
===

I have a lot of small objects in memory at the time of the exception but I'm
nowhere close to exhausting available RAM. Any ideas?

[Jon Skeet [C# MVP]]

On Oct 12, 2:03 pm, "Lamont Sanford" <yaBigDu...@san ford.sonwrote:

Can I suggest that you use perfmon to look at both the heap and the
handle count of your process?

Sure, I'll check it out. I've never used permon before. What signs/symptoms
should I be looking for?

Handles or heap memory rising constantly would be generally bad - but
you can also add performance traces for various .NET aspects, which
can be handy.

Jon

[Willy Denoyette [MVP]]

"Lamont Sanford" <ya********@san ford.sonwrote in message
news:MN******** *************** *******@giganew s.com...

>

>Maybe due to extensively allocating and freeing memory, your memory is
too much fragmentized ?

How would I go about defragmenting it? Should I manually run a garbage
collection?

No, there is nothing you can do about a fragmented heap (both managed and
process heap), other than restart the server.
Of course it's up to you to prevent possible heap fragmentation as much as
possible.
Therefore, I would like to know what version of the Framework and OS you
are running this on?
How large are the Lists in bytes, if you say up to 100k, do you mean 100kb
or something else, how did you measure this size?
How many of these list do you possibly create simultaneously?
Are you using Sockets connections?
Do you explicitly pin objects, or do you call into unmanaged code?

Willy.

[Zhangming Su]

I got the same problem before. Then I rewrited the application by replacing
DataSet by SqlDataReader.

"Lamont Sanford" <ya********@san ford.sonwrote in message
news:b_******** *************** *******@giganew s.com...

>I can't figure out why once or twice per week, my C# server application
throws OutOfMemoryExce ptions. There is plenty of memory in the machine. As
far as I can tell, when the exception is thrown, only 400 megs of 1gig
total is in use.

The exception is thrown while trying to update List<>. Here's a bit of the
stack trace:

===
System.OutOfMem oryException: Exception of type
'System.OutOfMe moryException' was thrown.
at System.Collecti ons.Generic.Lis t`1.set_Capacit y(Int32 value)
at System.Collecti ons.Generic.Lis t`1.EnsureCapac ity(Int32 min)
at System.Collecti ons.Generic.Lis t`1.Add(T item)
at Mosaic.Core.Sta tUtils.ExtractR ORs(MiniPerfIte m[] perfItems, Int32
startIdx, Int32 endIdx, Boolean useNAOnNonPosBa l) in
C:\Development\ Mosaic\Mosaic-AM\Mosaic.Core. Shared\Code\Uti lities\Statisti cal\StatUtils.c s:line
38
===

I have a lot of small objects in memory at the time of the exception but
I'm nowhere close to exhausting available RAM. Any ideas?

[not_a_commie]

Try using LinkedList<inst ead of List<if you can. It won't be as
efficient at stopping memory fragmentation, but it will not call the
EnsureCapacity function, which, in essence, momentarily triples the
amount of memory used so that it can double the amount of memory used.
Okay, so it's not really "memory used" unless you're using structs.
It's memory used for pointers, which on a 32bit system for 100k should
400kilobytes -- not much! (Tripling that should be fine which makes me
wonder if it's a list of structs.)

[Lamont Sanford]

Therefore, I would like to know what version of the Framework and OS you

are running this on?

I'm running .NET Framework 2.0 (VS 2005) on Win2003.

How large are the Lists in bytes, if you say up to 100k, do you mean 100kb
or something else, how did you measure this size?

I just meant that the size of the list (List<T>.Count) would never exceed
100,000. If type 'T' is a double and double requires 8 bytes, the total
required memory shouldn't exceed 800,000 bytes... and I have hundreds of
gigs free.

How many of these list do you possibly create simultaneously?

Only one can exist at a time, but these lists will be created over and over
again with different values.When a list is no longer needed, it isn't
explicitly Clear()'d, but it does fall out of scope so it should be GCd.

Are you using Sockets connections?

Yes, sockets are used. No more than 30 inbound connections are active at any
given time.

Do you explicitly pin objects, or do you call into unmanaged code?

What does "pin" mean? No, I don't make any calls into unmanaged code.

[Willy Denoyette [MVP]]

"Lamont Sanford" <ya********@san ford.sonwrote in message
news:dP******** *************** *******@giganew s.com...

>Therefore, I would like to know what version of the Framework and OS you
are running this on?

I'm running .NET Framework 2.0 (VS 2005) on Win2003.

>How large are the Lists in bytes, if you say up to 100k, do you mean
100kb or something else, how did you measure this size?

I just meant that the size of the list (List<T>.Count) would never exceed
100,000. If type 'T' is a double and double requires 8 bytes, the total
required memory shouldn't exceed 800,000 bytes... and I have hundreds of
gigs free.

No you don't? The virtual memory space attributed to your process is only
2GB (supposing W2K3 32 bit). That means that you cannot allocate more than
this space for managed objects minus the space allocated by the application
the runtime and the Framework data and code, this leaves you with less than
1.6GB (non contiguous) in the most optimal case at process creation time.
Now, 800 Kb is not a big deal, but as these objects are larger than 85Kb,
they end on the LOH and as this one is never compacted, you may end with a
highly fragmented heap, if you don't tput an eye on object life time.

>How many of these list do you possibly create simultaneously?

Only one can exist at a time, but these lists will be created over and
over again with different values.When a list is no longer needed, it isn't
explicitly Clear()'d, but it does fall out of scope so it should be GCd.

Yes, but that doesn't mean that only one can exist at a time, the GC doesn't
run deterministical ly.

>Are you using Sockets connections?

Yes, sockets are used. No more than 30 inbound connections are active at
any given time.

>Do you explicitly pin objects, or do you call into unmanaged code?

What does "pin" mean? No, I don't make any calls into unmanaged code.

Yes, you do (indirectly), there is no single managed application in the
world that don't call in unmanaged code. The socket classes need to pin the
send/reeive buffers when calling into the native Winsock API's. Pinning is
there to prevent the heap allocated object to move in memory when the GC
runs, however, pinning also prevents compactation of the GC heap.
Does it happen that these buffers have something in common with the above
mentioned List's?

Anyway, I would suggest you to inspect your memory allocation pattern by
running your code against a memory profiler.

Willy.

[Lamont Sanford]

>I just meant that the size of the list (List<T>.Count) would never exceed

>100,000. If type 'T' is a double and double requires 8 bytes, the total
required memory shouldn't exceed 800,000 bytes... and I have hundreds of
gigs free.

No you don't?

Ooops, I meant hundreds of megs, not hundreds of gigs... sorry for the
typeo.

Thanks for the hints. I'll take your advice and hopefully I can get to the
bottom of this.

[bob clegg]

Hi Lamont,
What is the Database engine that your app is interacting with?
Bob
On Thu, 11 Oct 2007 13:30:16 -0500, "Lamont Sanford"
<ya********@san ford.sonwrote:

>I can't figure out why once or twice per week, my C# server application
throws OutOfMemoryExce ptions. There is plenty of memory in the machine. As
far as I can tell, when the exception is thrown, only 400 megs of 1gig total
is in use.

The exception is thrown while trying to update List<>. Here's a bit of the
stack trace:

===
System.OutOfMe moryException: Exception of type 'System.OutOfMe moryException'
was thrown.
at System.Collecti ons.Generic.Lis t`1.set_Capacit y(Int32 value)
at System.Collecti ons.Generic.Lis t`1.EnsureCapac ity(Int32 min)
at System.Collecti ons.Generic.Lis t`1.Add(T item)
at Mosaic.Core.Sta tUtils.ExtractR ORs(MiniPerfIte m[] perfItems, Int32
startIdx, Int32 endIdx, Boolean useNAOnNonPosBa l) in
C:\Development \Mosaic\Mosaic-AM\Mosaic.Core. Shared\Code\Uti lities\Statisti cal\StatUtils.c s:line
38
===

I have a lot of small objects in memory at the time of the exception but I'm
nowhere close to exhausting available RAM. Any ideas?

[Lamont Sanford]

>

Hi Lamont,
What is the Database engine that your app is interacting with?
Bob

I'm using SQL Server 2005.

[bob]

OK.
Not what I thought.
I had weird problem with SQL Anywhere Version 9 that appeared as a
memory problem.
Sorry no other ideas.
Bob

On Wed, 17 Oct 2007 08:20:12 -0500, "Lamont Sanford"
<ya********@san ford.sonwrote:

>>
Hi Lamont,
What is the Database engine that your app is interacting with?
Bob

I'm using SQL Server 2005.