large ArrayList causes program to crash

[Stuart]

I have a small class of data values made up of ints and bools
totalling 25 bytes. Each of these 25 byte nodes is stored in another
class as an ArrayList with a few more ints. Typically there are around
400 nodes stored in each of these segment classes. - which makes each
segment occupy around 10KB.

I use further ArrayList to store around 30,000 of these segments,
which are read in from a file.

If a read in a file that contains 10,000 of these segments with the
node data, everything *seems* OK. Try reading 15,000 and the program
gets very slow at populating the ArrayList. Try the full 30,000, and
the program just goes on for ever trying to load the data in off the
disk, eventually warning me of low virtual memory.

The PC I'm developing on has 256MB of RAM. It looks to me as if too
much data is being stored to the stack instead of the heap, which I
would have thought would have been more than capable of storing 300MB
of segment data.

All I'm doing is creating instances of the classes, as the data is
read from the disk, then Adding them to the ArrayLists.

1) Does anyone have any advice as to if the ArrayList should be able
to handle this amount of data?
2) Are there any little gotchyas to watch out for when working with
large ArrayLists in c#?

Many thanks

Stuart

[David Browne]

"Stuart" <st*****@yahoo.com> wrote in message
news:d9**************************@posting.google.c om...

I have a small class of data values made up of ints and bools
totalling 25 bytes. Each of these 25 byte nodes is stored in another
class as an ArrayList with a few more ints. Typically there are around
400 nodes stored in each of these segment classes. - which makes each
segment occupy around 10KB.

I use further ArrayList to store around 30,000 of these segments,
which are read in from a file.

If a read in a file that contains 10,000 of these segments with the
node data, everything *seems* OK. Try reading 15,000 and the program
gets very slow at populating the ArrayList. Try the full 30,000, and
the program just goes on for ever trying to load the data in off the
disk, eventually warning me of low virtual memory.

The PC I'm developing on has 256MB of RAM. It looks to me as if too
much data is being stored to the stack instead of the heap, which I
would have thought would have been more than capable of storing 300MB
of segment data.

All I'm doing is creating instances of the classes, as the data is
read from the disk, then Adding them to the ArrayLists.

1) Does anyone have any advice as to if the ArrayList should be able
to handle this amount of data?
2) Are there any little gotchyas to watch out for when working with
large ArrayLists in c#?

Your managed heap will be at least 300mb large, but possibly larger
depending on how it gets collected. Then the memory required for the CLR
and the OS and other programs. I think 256M of ram is far too little to run
this program. My guess is that running this program and Visual Studio, you
will need total virtual memory of around 700 megs. For which you should
have at least 512meg of ram.

David

[Nicholas Paldino [.NET/C# MVP]]

Sutart,

I think that storing this much information is a little much. If
anything, would it be possible to store the data in a database of some sort,
where you could get what you need, instead of just loading everything at
once? Or rather, could you just load what you need in segments, and process
it and then dispose of it? It seems a little much to be loading all of this
at one time.

Hope this helps.
--
- Nicholas Paldino [.NET/C# MVP]
- mv*@spam.guard.caspershouse.com

"Stuart" <st*****@yahoo.com> wrote in message
news:d9**************************@posting.google.c om...

I have a small class of data values made up of ints and bools
totalling 25 bytes. Each of these 25 byte nodes is stored in another
class as an ArrayList with a few more ints. Typically there are around
400 nodes stored in each of these segment classes. - which makes each
segment occupy around 10KB.

I use further ArrayList to store around 30,000 of these segments,
which are read in from a file.

If a read in a file that contains 10,000 of these segments with the
node data, everything *seems* OK. Try reading 15,000 and the program
gets very slow at populating the ArrayList. Try the full 30,000, and
the program just goes on for ever trying to load the data in off the
disk, eventually warning me of low virtual memory.

The PC I'm developing on has 256MB of RAM. It looks to me as if too
much data is being stored to the stack instead of the heap, which I
would have thought would have been more than capable of storing 300MB
of segment data.

All I'm doing is creating instances of the classes, as the data is
read from the disk, then Adding them to the ArrayLists.

1) Does anyone have any advice as to if the ArrayList should be able
to handle this amount of data?
2) Are there any little gotchyas to watch out for when working with
large ArrayLists in c#?

Many thanks

Stuart

[Scott Allen]

Hi Stuart:

See inline:

On 30 Sep 2004 11:39:01 -0700, st*****@yahoo.com (Stuart) wrote:

I have a small class of data values made up of ints and bools
totalling 25 bytes. Each of these 25 byte nodes is stored in another
class as an ArrayList with a few more ints. Typically there are around
400 nodes stored in each of these segment classes. - which makes each
segment occupy around 10KB.

I use further ArrayList to store around 30,000 of these segments,
which are read in from a file.

If a read in a file that contains 10,000 of these segments with the
node data, everything *seems* OK. Try reading 15,000 and the program
gets very slow at populating the ArrayList. Try the full 30,000, and
the program just goes on for ever trying to load the data in off the
disk, eventually warning me of low virtual memory.

If the program exhausts physical memory, there is going to be a
performance impact. The OS will swap more pages in and out of RAM and
the system becomes disk-bound.
The PC I'm developing on has 256MB of RAM. It looks to me as if too
much data is being stored to the stack instead of the heap, which I
would have thought would have been more than capable of storing 300MB
of segment data.

If you are creating a new instance of a class, all the space allocated
is on the heap, even for the class fields which are value types like
int and bool.
All I'm doing is creating instances of the classes, as the data is
read from the disk, then Adding them to the ArrayLists.

1) Does anyone have any advice as to if the ArrayList should be able
to handle this amount of data?
The ArrayList can handle this amount of data, but I wouldn't try it on
a machine with 256MB of RAM. This is pretty unfriendly behavior to the
rest of the system even if more RAM is present. Is there any chance
you can do processing in smaller chunks?
2) Are there any little gotchyas to watch out for when working with
large ArrayLists in c#?

If the initial capacity of the ArrayList is not set, it defaults to a
capacity of 16. If a new node is added to the ArrayList and it needs
to expand, it will double capacity and the internal array is
reallocated. Reallocations are 'relatively' expensive, meaning it
probably won't make a big difference in your application anyway
because of the low memory situation.

--
Scott
http://www.OdeToCode.com/

[Stuart]

Thanks guys for the response!

I tested the application without the VS running on a machine 512MB at the
program ran correctly, albeit a little sluggish. I used CLR Profiler to
watch the data appearing on the heap too.

You have all confirmed one of the things I was thinking, in that there is
too much data in the application, even when it is all on the managed heap.

I will look to segmenting the data as suggested. I guess I could put it all
out to a temp file that I can access quickly as the application is running.
I think I have seen this sort of thing done with a commercial audio editing
program I have.

Thanks again.

Stuart
"Stuart" <st*****@yahoo.com> wrote in message
news:d9**************************@posting.google.c om...

I have a small class of data values made up of ints and bools
totalling 25 bytes. Each of these 25 byte nodes is stored in another
class as an ArrayList with a few more ints. Typically there are around
400 nodes stored in each of these segment classes. - which makes each
segment occupy around 10KB.

I use further ArrayList to store around 30,000 of these segments,
which are read in from a file.

If a read in a file that contains 10,000 of these segments with the
node data, everything *seems* OK. Try reading 15,000 and the program
gets very slow at populating the ArrayList. Try the full 30,000, and
the program just goes on for ever trying to load the data in off the
disk, eventually warning me of low virtual memory.

The PC I'm developing on has 256MB of RAM. It looks to me as if too
much data is being stored to the stack instead of the heap, which I
would have thought would have been more than capable of storing 300MB
of segment data.

All I'm doing is creating instances of the classes, as the data is
read from the disk, then Adding them to the ArrayLists.

1) Does anyone have any advice as to if the ArrayList should be able
to handle this amount of data?
2) Are there any little gotchyas to watch out for when working with
large ArrayLists in c#?

Many thanks

Stuart