473,287 Members | 1,581 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

Reading heavy .TXT file with StreamReader

I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog):

---------------------------------
System.IO.StreamReader sr = new System.IO.StreamReader(@"C:\1.txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace(",",".");
System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\1.txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object..........

Any help will be greatly appreciated.

Nov 17 '05 #1
13 3915
<ml*******@gmail.com> wrote:
[replacing characters in a 500 MB text file]
[...]
string all = sr.ReadToEnd();


This means you're reading the entire file into memory at once.
Instead, try having a StreamReader and a StreamWriter open at the same
time. Read a line from one file and write it to the other file, then
delete the first file when done and replace it with the second one.
You will only use the memory taken up by one line, rather than
slurping up 500 MB out of nowhere.

P.
Nov 17 '05 #2
Hi,
Below is a better solution:

string sourceFile = @"C:\1.txt";
string tempTarget = System.IO.Path.GetTempFileName();

StreamReader reader = new System.IO.StreamReader( sourceFile );
StreamWriter writer = new StreamWriter( tempTarget )

string line;
while ( (line = reader.ReadLine() )
writer.WriteLine( line.Replace( ",",".") )
reader.Close();
writer.Close();
File.Delete( source);
File.Copy( target, source);
cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
<ml*******@gmail.com> wrote in message
news:11**********************@f14g2000cwb.googlegr oups.com...
I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog):

---------------------------------
System.IO.StreamReader sr = new System.IO.StreamReader(@"C:\1.txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace(",",".");
System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\1.txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object..........

Any help will be greatly appreciated.

Nov 17 '05 #3
He probably wants to do a File.Move instead of a File.Copy for that last
one.

"Ignacio Machin ( .NET/ C# MVP )" <ignacio.machin AT dot.state.fl.us> wrote
in message news:Og****************@TK2MSFTNGP12.phx.gbl...
Hi,
Below is a better solution:

string sourceFile = @"C:\1.txt";
string tempTarget = System.IO.Path.GetTempFileName();

StreamReader reader = new System.IO.StreamReader( sourceFile );
StreamWriter writer = new StreamWriter( tempTarget )

string line;
while ( (line = reader.ReadLine() )
writer.WriteLine( line.Replace( ",",".") )
reader.Close();
writer.Close();
File.Delete( source);
File.Copy( target, source);
cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
<ml*******@gmail.com> wrote in message
news:11**********************@f14g2000cwb.googlegr oups.com...
I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog):

---------------------------------
System.IO.StreamReader sr = new System.IO.StreamReader(@"C:\1.txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace(",",".");
System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\1.txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object..........

Any help will be greatly appreciated.


Nov 17 '05 #4
Hi,

Yes, you are right :)

cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:2I********************@giganews.com...
He probably wants to do a File.Move instead of a File.Copy for that last
one.

"Ignacio Machin ( .NET/ C# MVP )" <ignacio.machin AT dot.state.fl.us>
wrote in message news:Og****************@TK2MSFTNGP12.phx.gbl...
Hi,
Below is a better solution:

string sourceFile = @"C:\1.txt";
string tempTarget = System.IO.Path.GetTempFileName();

StreamReader reader = new System.IO.StreamReader( sourceFile );
StreamWriter writer = new StreamWriter( tempTarget )

string line;
while ( (line = reader.ReadLine() )
writer.WriteLine( line.Replace( ",",".") )
reader.Close();
writer.Close();
File.Delete( source);
File.Copy( target, source);
cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
<ml*******@gmail.com> wrote in message
news:11**********************@f14g2000cwb.googlegr oups.com...
I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog):

---------------------------------
System.IO.StreamReader sr = new System.IO.StreamReader(@"C:\1.txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace(",",".");
System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\1.txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object..........

Any help will be greatly appreciated.



Nov 17 '05 #5
Thanks guys, specially Ignacio.

I'll try next days but really seems to be a good solution. If I have
any more problems, I'll return to the post.

Thanks again.

Nov 17 '05 #6
Hi again, the "while" statement does not compile because the while
comparison must return true and we are doing an assigment statement.
ReadLine() returns string and must be bool. Besides, I cannot do
while (reader.ReadLine() != String.Empty) because the file can contain
blank lines and this way we will go out of the while loop.

Any other idea?

Thanks.

Nov 17 '05 #7
It should be
while ( (line.reader.ReadLine()) != null)

Another suggestion, however - rather than having manual calls to Close,
you should use "using" statements round both the reader and the writer.
That way they automatically get closed whether an exception occurs or
not.

Jon

Nov 17 '05 #8
Hi guys,

The correct code, perfectly working, although I haven't tried with 500
MBytes .txt files:

-----------------------------------------------------------------------
string sourceFile = textBox1.Text;
string tempTarget = System.IO.Path.GetTempFileName();
StreamReader reader = new System.IO.StreamReader(sourceFile);
StreamWriter writer = new StreamWriter(tempTarget);

string line = String.Empty;
while ( (line = reader.ReadLine()) != null)
{
writer.WriteLine(line.Replace(txt1.Text,txt2.Text) );
}
reader.Close();
writer.Close();
File.Delete(sourceFile);
File.Move(tempTarget,sourceFile);
----------------------------------------------------------------------------

I'm not able to do it with "using" clause :(

Thanks Jon.

Nov 17 '05 #9
<ml*******@gmail.com> wrote:
Hi guys,

The correct code, perfectly working, although I haven't tried with 500
MBytes .txt files:

-----------------------------------------------------------------------
string sourceFile = textBox1.Text;
string tempTarget = System.IO.Path.GetTempFileName();
StreamReader reader = new System.IO.StreamReader(sourceFile);
StreamWriter writer = new StreamWriter(tempTarget);

string line = String.Empty;
while ( (line = reader.ReadLine()) != null)
{
writer.WriteLine(line.Replace(txt1.Text,txt2.Text) );
}
reader.Close();
writer.Close();
File.Delete(sourceFile);
File.Move(tempTarget,sourceFile);
----------------------------------------------------------------------------

I'm not able to do it with "using" clause :(


You should be able to:

using (StreamReader reader = new StreamReader (sourceFile))
{
using (StreamWriter writer = new StreamWriter (tempTarget))
{
string line;
while ( (line=reader.ReadLine()) != null)
{
writer.WriteLine (line.Replace(txt1.Text, txt2.text);
}
}
}
File.Delete(sourceFile);
File.Move(tempTarget,sourceFile);

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Nov 17 '05 #10
Congratulations for this simple but really good snipplet of code Jon.

Nested "Using" structure seems to be very comfortable and as you said,
you don't have to worry about closing neither the reader nor the
writer.

Thanks very much.

Regards,

Nov 17 '05 #11
Sorry to dissapoint you but my programming style goes more towards
Jon's style. Code looks like much more estructured and clearer for me.
I think in the example you only are "saving" two brackets, don't you?

One of my own programming rules: "It's better to have a clear and
well-structured code (brackets and so on...) in spite of having a
little bit more lines of code".

Thanks anyway. I didn't know you couldn't nest two "using" loops
without any brackets between them.

Regards.

Nov 17 '05 #12
Lonifasiko <ml*******@gmail.com> wrote:
Sorry to dissapoint you but my programming style goes more towards
Jon's style. Code looks like much more estructured and clearer for me.
I think in the example you only are "saving" two brackets, don't you?


It depends. You can easily get really heavily indented code with no
benefit in some situations - it's quite easy to end up with four nested
brackets without much in between.

I very rarely use the version with no extra brackets, but occasionally
it does improve readability. After all, you're treating the whole
contents as a single block.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Nov 17 '05 #13
> Sorry to dissapoint you but my programming style goes more towards
Jon's style. Code looks like much more estructured and clearer for me.
I think in the example you only are "saving" two brackets, don't you?
Thats ok, no dissapointment. What ever make you more productive.
One of my own programming rules: "It's better to have a clear and
well-structured code (brackets and so on...) in spite of having a
little bit more lines of code".


I would disagree with that. It is more clear, IMO, to do it the other way
as you see right away where they come in scope and leave scope at the same
place. For two streams it probably makes no difference. But for 3 or 4 (or
more), using the method I showed can really make it cleaner IMO. Your not
just saving the brackets, but your eliminating all the unnessasary nesting
and indents which can get distracting. And if your stacking streams in a
pipeline (i.e. file to compression to encryption to networkstream), it
really looks good. The end result is the same, but naturally everyone has
different tastes.

--
William Stacey [MVP]

Nov 17 '05 #14

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: jamait | last post by:
Hi all, I m trying to read in a text file into a datatable... Not sure on how to split up the information though, regex or substrings...? sample: Col1 Col2 ...
6
by: Neil Patel | last post by:
I have a log file that puts the most recent record at the bottom of the file. Each line is delimited by a \r\n Does anyone know how to seek to the end of the file and start reading backwards?
2
by: Sabin Finateanu | last post by:
Hi I'm having problem reading a file from my program and I think it's from a procedure I'm using but I don't see where I'm going wrong. Here is the code: public bool AllowUsage() { ...
9
by: haibhoang | last post by:
I have a Windows Service that is trying to parse a large (> 1Gig) text file. I am keep getting OutOfMemoryException exception. Here is the code that's having problem: using (StreamReader...
1
by: Gustaf | last post by:
I'm converting a program I made in Python once to C#, and while I'm at it, I want to do some performance improvements. The program takes a set of mailbox files (in readable formats, like Eudora and...
3
by: leticia larrosa | last post by:
Hi, I try to read a file that have 8 bit character, but contain some character whose code is more than 128 (spanish character). When I read this file using ASCII (Dim oRead As StreamReader =...
9
by: jeff M via .NET 247 | last post by:
I'm still having problems reading EBCDIC files. Currently itlooks like the lower range (0 to 127) is working. I have triedthe following code pages 20284, 20924, 1140, 37, 500 and 20127.By working I...
7
by: Drew Berkemeyer | last post by:
Hello, I'm using the following code to read a text file in VB.NET. Dim sr As StreamReader = File.OpenText(strFilePath) Dim input As String = sr.ReadLine() While Not input Is Nothing...
2
by: James Wong | last post by:
Dear all, I'm using StreamReader to read a text file containing BIG-5 data and found that no matter which encoding method in StreamReader's construction parameter, the BIG-5 contents become...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.