Why is OutOfMemoryException thrown?

Lamont Sanford

I can't figure out why once or twice per week, my C# server application
throws OutOfMemoryExceptions. There is plenty of memory in the machine. As
far as I can tell, when the exception is thrown, only 400 megs of 1gig total
is in use.

The exception is thrown while trying to update List<>. Here's a bit of the
stack trace:

===
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException'
was thrown.
at System.Collections.Generic.List`1.set_Capacity(Int 32 value)
at System.Collections.Generic.List`1.EnsureCapacity(I nt32 min)
at System.Collections.Generic.List`1.Add(T item)
at Mosaic.Core.StatUtils.ExtractRORs(MiniPerfItem[] perfItems, Int32
startIdx, Int32 endIdx, Boolean useNAOnNonPosBal) in
C:\Development\Mosaic\Mosaic-AM\Mosaic.Core.Shared\Code\Utilities\Statistical\S tatUtils.cs:line
38
===

I have a lot of small objects in memory at the time of the exception but I'm
nowhere close to exhausting available RAM. Any ideas?

Oct 11 '07 #1

Jon Skeet [C# MVP]

Lamont Sanford <yaBigDummy@sanford.sonwrote: Do you have any idea how big the list is at the point of failure?

--
Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Oct 11 '07 #2

Peter Morris

OutOfMemoryException really means "out of resources". Are you opening files
and not disposing them or something?

http://mrpmorris.blogspot.com/2006/1...maybe-not.html

Oct 11 '07 #3

Lamont Sanford

Do you have any idea how big the list is at the point of failure? At max it could get up to the tens of thousands, but certainly never over
100k...

Oct 11 '07 #4

Lamont Sanford

OutOfMemoryException really means "out of resources". Are you opening Interesting. I'll check out the article.

No, I don't make use of files to any major extent, but I do use a lot of
database connections. Maybe I'm not closing them properly?

Oct 11 '07 #5

=?Utf-8?B?UGV0ZXIgQnJvbWJlcmcgW0MjIE1WUF0=?=

Most likely - improperly closed / disposed connection objects means you run
out of Connection Pool connections, which can cause exceptions similar to
what you're seeing - especially if combined with other code.

1) use the same connection string througout your app.
2) open a connection just before you need to do database call.
3) close the connection immediately after - this returns it to the pool
4) be very careful with DataReaders as they hold open a connection.
-- Peter
Recursion: see Recursion
site: http://www.eggheadcafe.com
unBlog: http://petesbloggerama.blogspot.com
BlogMetaFinder: http://www.blogmetafinder.com



"Lamont Sanford" wrote:

Oct 11 '07 #6

Jon Skeet [C# MVP]

Lamont Sanford <yaBigDummy@sanford.sonwrote: That sounds unlikely to be the issue then.

Can I suggest that you use perfmon to look at both the heap and the
handle count of your process?

--
Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Oct 11 '07 #7

Przemek Celej

Lamont Sanford pisze: Maybe due to extensively allocating and freeing memory, your memory is
too much fragmentized ?

Regards

Oct 11 '07 #8

Lamont Sanford

> Sure, I'll check it out. I've never used permon before. What signs/symptoms
should I be looking for?

Oct 12 '07 #9

Lamont Sanford

How would I go about defragmenting it? Should I manually run a garbage
collection?

Oct 12 '07 #10

Jon Skeet [C# MVP]

On Oct 12, 2:03 pm, "Lamont Sanford" <yaBigDu...@sanford.sonwrote: Handles or heap memory rising constantly would be generally bad - but
you can also add performance traces for various .NET aspects, which
can be handy.

Jon

Oct 12 '07 #11

Willy Denoyette [MVP]

"Lamont Sanford" <yaBigDummy@sanford.sonwrote in message
news:MNWdnaM7ReBx7ZLanZ2dnUVZ_jSdnZ2d@giganews.com ...
No, there is nothing you can do about a fragmented heap (both managed and
process heap), other than restart the server.
Of course it's up to you to prevent possible heap fragmentation as much as
possible.
Therefore, I would like to know what version of the Framework and OS you
are running this on?
How large are the Lists in bytes, if you say up to 100k, do you mean 100kb
or something else, how did you measure this size?
How many of these list do you possibly create simultaneously?
Are you using Sockets connections?
Do you explicitly pin objects, or do you call into unmanaged code?

Willy.

Oct 12 '07 #12

Zhangming Su

I got the same problem before. Then I rewrited the application by replacing
DataSet by SqlDataReader.

"Lamont Sanford" <yaBigDummy@sanford.sonwrote in message
news:b_WdnUmyg6El9pPanZ2dnUVZ_uWlnZ2d@giganews.com ...

Oct 12 '07 #13

not_a_commie

Try using LinkedList<instead of List<if you can. It won't be as
efficient at stopping memory fragmentation, but it will not call the
EnsureCapacity function, which, in essence, momentarily triples the
amount of memory used so that it can double the amount of memory used.
Okay, so it's not really "memory used" unless you're using structs.
It's memory used for pointers, which on a 32bit system for 100k should
400kilobytes -- not much! (Tripling that should be fine which makes me
wonder if it's a list of structs.)

Oct 12 '07 #14

Lamont Sanford

Therefore, I would like to know what version of the Framework and OS you I'm running .NET Framework 2.0 (VS 2005) on Win2003.
I just meant that the size of the list (List<T>.Count) would never exceed
100,000. If type 'T' is a double and double requires 8 bytes, the total
required memory shouldn't exceed 800,000 bytes... and I have hundreds of
gigs free.
Only one can exist at a time, but these lists will be created over and over
again with different values.When a list is no longer needed, it isn't
explicitly Clear()'d, but it does fall out of scope so it should be GCd.
Yes, sockets are used. No more than 30 inbound connections are active at any
given time.
What does "pin" mean? No, I don't make any calls into unmanaged code.

Oct 15 '07 #15

Willy Denoyette [MVP]

"Lamont Sanford" <yaBigDummy@sanford.sonwrote in message
news:dP-dnSP5rd3T9I7anZ2dnUVZ_v-hnZ2d@giganews.com... No you don't? The virtual memory space attributed to your process is only
2GB (supposing W2K3 32 bit). That means that you cannot allocate more than
this space for managed objects minus the space allocated by the application
the runtime and the Framework data and code, this leaves you with less than
1.6GB (non contiguous) in the most optimal case at process creation time.
Now, 800 Kb is not a big deal, but as these objects are larger than 85Kb,
they end on the LOH and as this one is never compacted, you may end with a
highly fragmented heap, if you don't tput an eye on object life time.
Yes, but that doesn't mean that only one can exist at a time, the GC doesn't
run deterministically.
Yes, you do (indirectly), there is no single managed application in the
world that don't call in unmanaged code. The socket classes need to pin the
send/reeive buffers when calling into the native Winsock API's. Pinning is
there to prevent the heap allocated object to move in memory when the GC
runs, however, pinning also prevents compactation of the GC heap.
Does it happen that these buffers have something in common with the above
mentioned List's?

Anyway, I would suggest you to inspect your memory allocation pattern by
running your code against a memory profiler.

Willy.

Oct 15 '07 #16

Lamont Sanford

>I just meant that the size of the list (List<T>.Count) would never exceed Ooops, I meant hundreds of megs, not hundreds of gigs... sorry for the
typeo.

Thanks for the hints. I'll take your advice and hopefully I can get to the
bottom of this.

Oct 15 '07 #17