memory leak with streamwriter? help!!

[Rob Corwin]

Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob

[Eric Gunnerson [MS]]

Rob,

Have you tried calling Flush() on it?

I've added the frameworks newsgroup to the discussion.

--
Eric Gunnerson

Visit the C# product team at http://www.csharp.net
Eric's blog is at http://blogs.gotdotnet.com/ericgu/

This posting is provided "AS IS" with no warranties, and confers no rights.
"Rob Corwin" <co****@haas.berkeley.edu> wrote in message
news:O8**************@TK2MSFTNGP09.phx.gbl...

Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob

[Jon Skeet [C# MVP]]

Rob Corwin <co****@haas.berkeley.edu> wrote:

a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

<snip>

Could you post a short but complete example which demonstrates the
problem?

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Alvin Bruney]

I'm told by one tech ed instructor that this is a framework 1.x bug which is
slated for correction in 2.0.

I've been looking into similar behavior with memory consumption of a very
large data set with the exact same behavior for a couple weeks now and that
is the latest I have on this issue. Nulling, flushing, calling garbage
collection doesn't help. The memory climbs and eventually either causes an
application recycle or a hard crash.

--
-----------
Got TidBits?
Get it here: www.networkip.net/tidbits
"Rob Corwin" <co****@haas.berkeley.edu> wrote in message
news:O8**************@TK2MSFTNGP09.phx.gbl...

Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob

[Willy Denoyette [MVP]]

I suppose x is a string variable, what's the size of the string?
What version of the runtime are you running?

Willy.
"Rob Corwin" <co****@haas.berkeley.edu> wrote in message news:O8**************@TK2MSFTNGP09.phx.gbl...

Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob

[Robert Corwin]

hi guys,
thanks for all of your replies. I think that this problem is probably
similar to the one that Alvin has encountered, so I may just have to
find a workaround. However in response to your questions, I've tried
many of the streamwriter methods (flush etc) and GC methods (collect
etc) and, like Alvin's problem, nulling, collecting, etc. doesn't help.

if you guys know how to destory memory manually or have any other types
of memory debugging tools it would be much appreciated... but in the
meantime I am going to code around this somehow.

thanks again
rob
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

[Robert Corwin]

FYI - I found out that this is a known bug in .NET 1.0 and 1.1, and that
it is unlikely that there will be a service pack to correct this before
2.0 comes out.

regards,
Rob
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

[Willy Denoyette [MVP]]

Large object allocations (>85Kb) require contiguous blocks of memory, when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1 GC can't handle yet.

Willy.

"Robert Corwin" <co****@haas.berkeley.edu> wrote in message news:uh**************@TK2MSFTNGP10.phx.gbl...

hi guys,
thanks for all of your replies. I think that this problem is probably
similar to the one that Alvin has encountered, so I may just have to
find a workaround. However in response to your questions, I've tried
many of the streamwriter methods (flush etc) and GC methods (collect
etc) and, like Alvin's problem, nulling, collecting, etc. doesn't help.

if you guys know how to destory memory manually or have any other types
of memory debugging tools it would be much appreciated... but in the
meantime I am going to code around this somehow.

thanks again
rob
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

[Jon Skeet [C# MVP]]

Willy Denoyette [MVP] <wi*************@pandora.be> wrote:

Large object allocations (>85Kb) require contiguous blocks of memory,
when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1
GC can't handle yet.

That was definitely a problem in 1.0, but I thought it was fixed with a
service pack and definitely fixed by 1.1.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Robert Corwin]

don't think this is an issue with the 85K object size, or irregular
allocation.

I've had 2 people tell me that there is a separate bug with streamwriter
and some DataSets - memory leaks under certain circumstances when you
are dealing with large amounts of data/memory.
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

[Willy Denoyette [MVP]]

Jon,

There are two different bug related to large object heap exhaustion.
The first was corrected in v1.0 SP2 and v1.1, but 1.1 still doesn't handle heap fragmentation correctly (v2.0 will definitely handle
this), I guess a SP for v1.1 will be released before v2.0 (RTM).

Willy.
"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message news:MP************************@msnews.microsoft.c om...

Willy Denoyette [MVP] <wi*************@pandora.be> wrote:

Large object allocations (>85Kb) require contiguous blocks of memory,
when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1
GC can't handle yet.

That was definitely a problem in 1.0, but I thought it was fixed with a
service pack and definitely fixed by 1.1.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Willy Denoyette [MVP]]

"Robert Corwin" <co****@haas.berkeley.edu> wrote in message news:e6**************@TK2MSFTNGP12.phx.gbl...

don't think this is an issue with the 85K object size, or irregular
allocation.
are dealing with large amounts of data/memory.
Yes, large strings (>85 kb) of irregular size f.i.

Willy.

*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

[Alvin Bruney]

Willy:
I'm getting confused with this heap fragmentation stuff. The .NET memory
model should not be causing heap fragmentation in managed code because of
how memory is allocated and initialized (pointer plus offset). Otherwise I
need to unlearn some stuff and go yell at Richter.

Garbage collection should not intefer with this either. So where is the
'fragmentation' coming from? Just to be clear, we are talking about managed
memory right?

Large object allocations (>85Kb) require contiguous blocks of memory,
when the allocation scheme is irregular

That's what confuses me. It doesn't seem to be correct and you need to
justify this statement because memory is initialized long before the
application runs into contiguous blocks. Large blocks should be allocated
the same way smaller blocks are - pointer plus offset. Where is the
irregularity there?

regards
--
-----------
Got TidBits?
Get it here: www.networkip.net/tidbits
"Willy Denoyette [MVP]" <wi*************@pandora.be> wrote in message
news:O4**************@TK2MSFTNGP10.phx.gbl...

Jon,

There are two different bug related to large object heap exhaustion.
The first was corrected in v1.0 SP2 and v1.1, but 1.1 still doesn't handle heap fragmentation correctly (v2.0 will definitely handle this), I guess a SP for v1.1 will be released before v2.0 (RTM).

Willy.
"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message

news:MP************************@msnews.microsoft.c om... Willy Denoyette [MVP] <wi*************@pandora.be> wrote:

Large object allocations (>85Kb) require contiguous blocks of memory,
when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1
GC can't handle yet.

That was definitely a problem in 1.0, but I thought it was fixed with a
service pack and definitely fixed by 1.1.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Alan Pretre]

"Willy Denoyette [MVP]" <wi*************@pandora.be> wrote in message
news:O4**************@TK2MSFTNGP10.phx.gbl...

There are two different bug related to large object heap exhaustion.
The first was corrected in v1.0 SP2 and v1.1, but 1.1 still doesn't handle heap fragmentation correctly (v2.0 will definitely handle this), I guess a SP for v1.1 will be released before v2.0 (RTM).

If it's just a bug fix, it's hard to see why this should take so long. If
it's an algorithm change, I guess it makes sense. But it sounds like junior
year comp sci stuff to me...

-- Alan