473,836 Members | 1,593 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

memory leak with streamwriter? help!!

Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileS ystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!! !!!
thanks in advance
Rob

Nov 15 '05 #1
13 5719
Rob,

Have you tried calling Flush() on it?

I've added the frameworks newsgroup to the discussion.

--
Eric Gunnerson

Visit the C# product team at http://www.csharp.net
Eric's blog is at http://blogs.gotdotnet.com/ericgu/

This posting is provided "AS IS" with no warranties, and confers no rights.
"Rob Corwin" <co****@haas.be rkeley.edu> wrote in message
news:O8******** ******@TK2MSFTN GP09.phx.gbl...
Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileS ystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!! !!!
thanks in advance
Rob

Nov 15 '05 #2
Rob Corwin <co****@haas.be rkeley.edu> wrote:
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:


<snip>

Could you post a short but complete example which demonstrates the
problem?

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 15 '05 #3
I'm told by one tech ed instructor that this is a framework 1.x bug which is
slated for correction in 2.0.

I've been looking into similar behavior with memory consumption of a very
large data set with the exact same behavior for a couple weeks now and that
is the latest I have on this issue. Nulling, flushing, calling garbage
collection doesn't help. The memory climbs and eventually either causes an
application recycle or a hard crash.

--
-----------
Got TidBits?
Get it here: www.networkip.net/tidbits
"Rob Corwin" <co****@haas.be rkeley.edu> wrote in message
news:O8******** ******@TK2MSFTN GP09.phx.gbl...
Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileS ystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!! !!!
thanks in advance
Rob

Nov 15 '05 #4
I suppose x is a string variable, what's the size of the string?
What version of the runtime are you running?

Willy.
"Rob Corwin" <co****@haas.be rkeley.edu> wrote in message news:O8******** ******@TK2MSFTN GP09.phx.gbl...
Hi,
a c# app of mine that parses 30,000 xml files writes large amounts of
data to file (> 2GB) using a streamwriter object (sw). everything works
fine except that the memory used by the app grows linearly during
execution and eventually crashes the computer. without going into too
much detail of my code, i made the following observations:

- if you comment out the sw.Write(x) statement (which is inside the loop
that parses the xml files), memory doesn't increase. thus it must be
something to do with the streamwriter or file I/O.

- if you change x in sw.Write(x) to be a constant, memory doesn't
increase, but if x is an expression or variable, memory does increase.
Not sure what this means.

I've tried many things to solve this problem and am completely stuck.
I've tried making sw = null every so often within the loop, making a new
streamwriter and output file (so that the file sizes never exceed 10MB),
and calling GC.Collect() to try and force the compiler to clean up
memory, but there is no effect. I've tried using{} statements which
don't do anything either. The only thing I can think of is to re-run
the entire application multiple times and pass command line parameters
to indicate where the last one left off, but this is obviously not an
ideal solution.

I did get it to work by using the Scripting.FileS ystemObejct via COM
instead of streamwriter, but it's prohibitively slow. However this does
indicate that there are no memory leaks within the rest of my code, and
that the problem is with streamwriter somehow.

Any thoughts? Any help at all is greatly appreciated!!!! !!!
thanks in advance
Rob

Nov 15 '05 #5


hi guys,
thanks for all of your replies. I think that this problem is probably
similar to the one that Alvin has encountered, so I may just have to
find a workaround. However in response to your questions, I've tried
many of the streamwriter methods (flush etc) and GC methods (collect
etc) and, like Alvin's problem, nulling, collecting, etc. doesn't help.

if you guys know how to destory memory manually or have any other types
of memory debugging tools it would be much appreciated... but in the
meantime I am going to code around this somehow.

thanks again
rob
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!
Nov 15 '05 #6


FYI - I found out that this is a known bug in .NET 1.0 and 1.1, and that
it is unlikely that there will be a service pack to correct this before
2.0 comes out.

regards,
Rob
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!
Nov 15 '05 #7
Large object allocations (>85Kb) require contiguous blocks of memory, when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1 GC can't handle yet.

Willy.

"Robert Corwin" <co****@haas.be rkeley.edu> wrote in message news:uh******** ******@TK2MSFTN GP10.phx.gbl...


hi guys,
thanks for all of your replies. I think that this problem is probably
similar to the one that Alvin has encountered, so I may just have to
find a workaround. However in response to your questions, I've tried
many of the streamwriter methods (flush etc) and GC methods (collect
etc) and, like Alvin's problem, nulling, collecting, etc. doesn't help.

if you guys know how to destory memory manually or have any other types
of memory debugging tools it would be much appreciated... but in the
meantime I am going to code around this somehow.

thanks again
rob
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!

Nov 15 '05 #8
Willy Denoyette [MVP] <wi************ *@pandora.be> wrote:
Large object allocations (>85Kb) require contiguous blocks of memory,
when the allocation scheme is irregular, allocated memory tend
to grow, this seems to be one of the problems the version 1.0 and 1.1
GC can't handle yet.


That was definitely a problem in 1.0, but I thought it was fixed with a
service pack and definitely fixed by 1.1.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 15 '05 #9


don't think this is an issue with the 85K object size, or irregular
allocation.

I've had 2 people tell me that there is a separate bug with streamwriter
and some DataSets - memory leaks under certain circumstances when you
are dealing with large amounts of data/memory.
*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!
Nov 15 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
4677
by: Jeremy Lemaire | last post by:
Hello, I am working on cross platform code that is displaying a huge memory leak when compiled on 11.00 HPUX using the aCC -AA flag. It is not leaking on NT, LINUX, Solaris, or HPUX without the -AA flag. In another news group I came across some interesting (ok scarey) information regarding memory leaks in the STL list<...> container. I have compiled and executed the following code and verified that this does in fact leak on my system.
6
2572
by: Scott Niu | last post by:
Hi, I have this following simple c++ program, it will produce memory leak ( see what I did below ). My observation also showed that: There will be a mem leak when all the 3 conditions are true: 1. CallControlData does not provide a operator= 2. INUserHandle's operator= returns a object instead of a reference 3. Use CC -g main_leak.C instead of CC main_leak.C to compile. You change any of the above condition, there will be NO mem leak.
4
2355
by: Morten Aune Lyrstad | last post by:
Ok, now I'm officially confused. I have a large project going, which uses a win32 ui library I am developing myself. And I'm getting weird memory leaks. I don't know if I can explain what is going on, but I really need some help on this one. Ok, so I have this class defined (written by Randy Charles Morin, www.kbcafe.com) which detects memory leaks. It creates a memory check point in the constructor, and another in the destructor, and...
2
1610
by: tuarek | last post by:
Hi all, Our DB2 Server processes used up host machine's whole memory. Without getting the server down, how can we relieve the memory load. CRM Users are getting error messages, I need something quick & dirty. Thanks in advance...
4
6095
by: Don Nell | last post by:
Hello Why is there a memory leak when this code is executed. for(;;) { ManagementScope scope = new ManagementScope(); scope.Options.Username="username"; scope.Options.Password="password"; scope.Path.Path=@"\\pc\root\cimv2";
3
1735
by: Chris Botha | last post by:
I have a Windows app (it will be promoted to a Service once this problem is sorted out) that cyclically, every minute, connects to five SQL Servers over the WAN, extract data and insert the data in a local SQL Server, using the OLEDB provider. Every cycle I create one local connection and five remote connections, do the stuff, close the connections and dispose everything that has a Dispose method (OleDbConnection, OleDbDataAdapter and...
30
4709
by: MAG1301 | last post by:
I've detected memory leaks in our huge .NET 1.1 C# application but couldn't localize them directly. So I've reduced the code to the following console application: using System; using System.IO; namespace MemLeak { class MemLeak
3
3382
by: KWienhold | last post by:
I'm currently writing an application (using Visual Studio 2003 SP1 and C#) that stores files and additional information in a single compound file using IStorage/IStream. Since files in a compound file aren't really useful to a user, I use the IStream::Read function together with a StreamWriter to extract single files from my compound document. When I first tested these functions everything seemed to work fine (and basically, it does),...
0
1677
by: nejucomo | last post by:
Hi folks, Quick Synopsis: A test script demonstrates a memory leak when I use pythonic extensions of my builtin types, but if I use the builtin types themselves there is no memory leak. If you are interested in how builtin/pure-python inheritance interacts
3
2928
by: talefsa | last post by:
Hi, I'm trying to figure out the origin of a memory corruption in my code using valgrind. valgrind gave me a number of messages that didn't help much, the first invalid read/write message is as follow: ==22774== Invalid write of size 4 ==22774== at 0x488F53A: std::string::_Alloc_hider::_Alloc_hider(char*, std::allocator<char> const&) (in /usr/lib/libstdc++.so.5.0.3) ==22774== by 0x488FC26: std::string::string(char const*,...
0
9825
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
10859
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10260
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7795
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
6984
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5829
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4463
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4023
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
3116
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.