468,791 Members | 1,781 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,791 developers. It's quick & easy.

memory leak with streamwriter? help!!

Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob

Nov 15 '05 #1
13 4861
Rob,

Have you tried calling Flush() on it?

I've added the frameworks newsgroup to the discussion.

--
Eric Gunnerson

Visit the C# product team at http://www.csharp.net
Eric's blog is at http://blogs.gotdotnet.com/ericgu/

This posting is provided "AS IS" with no warranties, and confers no rights.
"Rob Corwin" <co****@haas.berkeley.edu> wrote in message
news:O8**************@TK2MSFTNGP09.phx.gbl...
Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob

Nov 15 '05 #2
Rob Corwin <co****@haas.berkeley.edu> wrote:
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:


<snip>

Could you post a short but complete example which demonstrates the
problem?

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 15 '05 #3
I'm told by one tech ed instructor that this is a framework 1.x bug which is
slated for correction in 2.0.

I've been looking into similar behavior with memory consumption of a very
large data set with the exact same behavior for a couple weeks now and that
is the latest I have on this issue. Nulling, flushing, calling garbage
collection doesn't help. The memory climbs and eventually either causes an
application recycle or a hard crash.

--
-----------
Got TidBits?
Get it here: www.networkip.net/tidbits
"Rob Corwin" <co****@haas.berkeley.edu> wrote in message
news:O8**************@TK2MSFTNGP09.phx.gbl...
Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob

Nov 15 '05 #4
I suppose x is a string variable, what's the size of the string?
What version of the runtime are you running?

Willy.
"Rob Corwin" <co****@haas.berkeley.edu> wrote in message news:O8**************@TK2MSFTNGP09.phx.gbl...
Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileSystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!!!!!
thanks in advance
Rob

Nov 15 '05 #5


hi guys,
thanks for all of your replies. I think that this problem is probably
similar to the one that Alvin has encountered, so I may just have to
find a workaround. However in response to your questions, I've tried
many of the streamwriter methods (flush etc) and GC methods (collect
etc) and, like Alvin's problem, nulling, collecting, etc. doesn't help.

if you guys know how to destory memory manually or have any other types
of memory debugging tools it would be much appreciated... but in the
meantime I am going to code around this somehow.

thanks again
rob
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!
Nov 15 '05 #6


FYI - I found out that this is a known bug in .NET 1.0 and 1.1, and that
it is unlikely that there will be a service pack to correct this before
2.0 comes out.

regards,
Rob
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!
Nov 15 '05 #7
Large object allocations (>85Kb) require contiguous blocks of memory, when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1 GC can't handle yet.

Willy.

"Robert Corwin" <co****@haas.berkeley.edu> wrote in message news:uh**************@TK2MSFTNGP10.phx.gbl...


hi guys,
thanks for all of your replies. I think that this problem is probably
similar to the one that Alvin has encountered, so I may just have to
find a workaround. However in response to your questions, I've tried
many of the streamwriter methods (flush etc) and GC methods (collect
etc) and, like Alvin's problem, nulling, collecting, etc. doesn't help.

if you guys know how to destory memory manually or have any other types
of memory debugging tools it would be much appreciated... but in the
meantime I am going to code around this somehow.

thanks again
rob
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

Nov 15 '05 #8
Willy Denoyette [MVP] <wi*************@pandora.be> wrote:
Large object allocations (>85Kb) require contiguous blocks of memory,
when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1
GC can't handle yet.


That was definitely a problem in 1.0, but I thought it was fixed with a
service pack and definitely fixed by 1.1.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 15 '05 #9


don't think this is an issue with the 85K object size, or irregular
allocation.

I've had 2 people tell me that there is a separate bug with streamwriter
and some DataSets - memory leaks under certain circumstances when you
are dealing with large amounts of data/memory.
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!
Nov 15 '05 #10
Jon,

There are two different bug related to large object heap exhaustion.
The first was corrected in v1.0 SP2 and v1.1, but 1.1 still doesn't handle heap fragmentation correctly (v2.0 will definitely handle
this), I guess a SP for v1.1 will be released before v2.0 (RTM).

Willy.
"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message news:MP************************@msnews.microsoft.c om...
Willy Denoyette [MVP] <wi*************@pandora.be> wrote:
Large object allocations (>85Kb) require contiguous blocks of memory,
when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1
GC can't handle yet.


That was definitely a problem in 1.0, but I thought it was fixed with a
service pack and definitely fixed by 1.1.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Nov 15 '05 #11

"Robert Corwin" <co****@haas.berkeley.edu> wrote in message news:e6**************@TK2MSFTNGP12.phx.gbl...


don't think this is an issue with the 85K object size, or irregular
allocation.
are dealing with large amounts of data/memory.
Yes, large strings (>85 kb) of irregular size f.i.

Willy.

*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

Nov 15 '05 #12
Willy:
I'm getting confused with this heap fragmentation stuff. The .NET memory
model should not be causing heap fragmentation in managed code because of
how memory is allocated and initialized (pointer plus offset). Otherwise I
need to unlearn some stuff and go yell at Richter.

Garbage collection should not intefer with this either. So where is the
'fragmentation' coming from? Just to be clear, we are talking about managed
memory right?
Large object allocations (>85Kb) require contiguous blocks of memory,
when the allocation scheme is irregular

That's what confuses me. It doesn't seem to be correct and you need to
justify this statement because memory is initialized long before the
application runs into contiguous blocks. Large blocks should be allocated
the same way smaller blocks are - pointer plus offset. Where is the
irregularity there?

regards
--
-----------
Got TidBits?
Get it here: www.networkip.net/tidbits
"Willy Denoyette [MVP]" <wi*************@pandora.be> wrote in message
news:O4**************@TK2MSFTNGP10.phx.gbl...
Jon,

There are two different bug related to large object heap exhaustion.
The first was corrected in v1.0 SP2 and v1.1, but 1.1 still doesn't handle heap fragmentation correctly (v2.0 will definitely handle this), I guess a SP for v1.1 will be released before v2.0 (RTM).

Willy.
"Jon Skeet [C# MVP]" <sk***@pobox.com> wrote in message

news:MP************************@msnews.microsoft.c om... Willy Denoyette [MVP] <wi*************@pandora.be> wrote:
Large object allocations (>85Kb) require contiguous blocks of memory,
when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1
GC can't handle yet.


That was definitely a problem in 1.0, but I thought it was fixed with a
service pack and definitely fixed by 1.1.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too


Nov 15 '05 #13
"Willy Denoyette [MVP]" <wi*************@pandora.be> wrote in message
news:O4**************@TK2MSFTNGP10.phx.gbl...
There are two different bug related to large object heap exhaustion.
The first was corrected in v1.0 SP2 and v1.1, but 1.1 still doesn't handle heap fragmentation correctly (v2.0 will definitely handle this), I guess a SP for v1.1 will be released before v2.0 (RTM).


If it's just a bug fix, it's hard to see why this should take so long. If
it's an algorithm change, I guess it makes sense. But it sounds like junior
year comp sci stuff to me...

-- Alan
Nov 15 '05 #14

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

4 posts views Thread by Morten Aune Lyrstad | last post: by
2 posts views Thread by tuarek | last post: by
4 posts views Thread by Don Nell | last post: by
3 posts views Thread by Chris Botha | last post: by
30 posts views Thread by MAG1301 | last post: by
3 posts views Thread by KWienhold | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
2 posts views Thread by Marin | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.