Reading heavy .TXT file with StreamReader

[mloichate]

I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog) :

---------------------------------
System.IO.Strea mReader sr = new System.IO.Strea mReader(@"C:\1. txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace("," ,".");
System.IO.Strea mWriter sw = new System.IO.Strea mWriter(@"C:\1. txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show ("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object......... .

Any help will be greatly appreciated.

[Paul E Collins]

<ml*******@gmai l.com> wrote:

[replacing characters in a 500 MB text file]
[...]
string all = sr.ReadToEnd();

This means you're reading the entire file into memory at once.
Instead, try having a StreamReader and a StreamWriter open at the same
time. Read a line from one file and write it to the other file, then
delete the first file when done and replace it with the second one.
You will only use the memory taken up by one line, rather than
slurping up 500 MB out of nowhere.

P.

[Ignacio Machin \( .NET/ C# MVP \)]

Hi,
Below is a better solution:

string sourceFile = @"C:\1.txt";
string tempTarget = System.IO.Path. GetTempFileName ();

StreamReader reader = new System.IO.Strea mReader( sourceFile );
StreamWriter writer = new StreamWriter( tempTarget )

string line;
while ( (line = reader.ReadLine () )
writer.WriteLin e( line.Replace( ",",".") )
reader.Close();
writer.Close();
File.Delete( source);
File.Copy( target, source);
cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
<ml*******@gmai l.com> wrote in message
news:11******** **************@ f14g2000cwb.goo glegroups.com.. .

I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog) :

---------------------------------
System.IO.Strea mReader sr = new System.IO.Strea mReader(@"C:\1. txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace("," ,".");
System.IO.Strea mWriter sw = new System.IO.Strea mWriter(@"C:\1. txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show ("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object......... .

Any help will be greatly appreciated.

[Pete Davis]

He probably wants to do a File.Move instead of a File.Copy for that last
one.

"Ignacio Machin ( .NET/ C# MVP )" <ignacio.mach in AT dot.state.fl.us > wrote
in message news:Og******** ********@TK2MSF TNGP12.phx.gbl. ..

Hi,
Below is a better solution:

string sourceFile = @"C:\1.txt";
string tempTarget = System.IO.Path. GetTempFileName ();

StreamReader reader = new System.IO.Strea mReader( sourceFile );
StreamWriter writer = new StreamWriter( tempTarget )

string line;
while ( (line = reader.ReadLine () )
writer.WriteLin e( line.Replace( ",",".") )
reader.Close();
writer.Close();
File.Delete( source);
File.Copy( target, source);
cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
<ml*******@gmai l.com> wrote in message
news:11******** **************@ f14g2000cwb.goo glegroups.com.. .

I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog) :

---------------------------------
System.IO.Strea mReader sr = new System.IO.Strea mReader(@"C:\1. txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace("," ,".");
System.IO.Strea mWriter sw = new System.IO.Strea mWriter(@"C:\1. txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show ("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object......... .

Any help will be greatly appreciated.

[Ignacio Machin \( .NET/ C# MVP \)]

Hi,

Yes, you are right :)

cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:2I******** ************@gi ganews.com...

He probably wants to do a File.Move instead of a File.Copy for that last
one.

"Ignacio Machin ( .NET/ C# MVP )" <ignacio.mach in AT dot.state.fl.us >
wrote in message news:Og******** ********@TK2MSF TNGP12.phx.gbl. ..

Hi,
Below is a better solution:

string sourceFile = @"C:\1.txt";
string tempTarget = System.IO.Path. GetTempFileName ();

StreamReader reader = new System.IO.Strea mReader( sourceFile );
StreamWriter writer = new StreamWriter( tempTarget )

string line;
while ( (line = reader.ReadLine () )
writer.WriteLin e( line.Replace( ",",".") )
reader.Close();
writer.Close();
File.Delete( source);
File.Copy( target, source);
cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
<ml*******@gmai l.com> wrote in message
news:11******** **************@ f14g2000cwb.goo glegroups.com.. .

I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog) :

---------------------------------
System.IO.Strea mReader sr = new System.IO.Strea mReader(@"C:\1. txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace("," ,".");
System.IO.Strea mWriter sw = new System.IO.Strea mWriter(@"C:\1. txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show ("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object......... .

Any help will be greatly appreciated.

[mloichate]

Thanks guys, specially Ignacio.

I'll try next days but really seems to be a good solution. If I have
any more problems, I'll return to the post.

Thanks again.

[mloichate]

Hi again, the "while" statement does not compile because the while
comparison must return true and we are doing an assigment statement.
ReadLine() returns string and must be bool. Besides, I cannot do
while (reader.ReadLin e() != String.Empty) because the file can contain
blank lines and this way we will go out of the while loop.

Any other idea?

Thanks.

[Jon Skeet [C# MVP]]

It should be
while ( (line.reader.Re adLine()) != null)

Another suggestion, however - rather than having manual calls to Close,
you should use "using" statements round both the reader and the writer.
That way they automatically get closed whether an exception occurs or
not.

Jon

[mloichate]

Hi guys,

The correct code, perfectly working, although I haven't tried with 500
MBytes .txt files:

-----------------------------------------------------------------------
string sourceFile = textBox1.Text;
string tempTarget = System.IO.Path. GetTempFileName ();
StreamReader reader = new System.IO.Strea mReader(sourceF ile);
StreamWriter writer = new StreamWriter(te mpTarget);

string line = String.Empty;
while ( (line = reader.ReadLine ()) != null)
{
writer.WriteLin e(line.Replace( txt1.Text,txt2. Text));
}
reader.Close();
writer.Close();
File.Delete(sou rceFile);
File.Move(tempT arget,sourceFil e);
----------------------------------------------------------------------------

I'm not able to do it with "using" clause :(

Thanks Jon.

[Jon Skeet [C# MVP]]

<ml*******@gmai l.com> wrote:

Hi guys,

The correct code, perfectly working, although I haven't tried with 500
MBytes .txt files:

-----------------------------------------------------------------------
string sourceFile = textBox1.Text;
string tempTarget = System.IO.Path. GetTempFileName ();
StreamReader reader = new System.IO.Strea mReader(sourceF ile);
StreamWriter writer = new StreamWriter(te mpTarget);

string line = String.Empty;
while ( (line = reader.ReadLine ()) != null)
{
writer.WriteLin e(line.Replace( txt1.Text,txt2. Text));
}
reader.Close();
writer.Close();
File.Delete(sou rceFile);
File.Move(tempT arget,sourceFil e);
----------------------------------------------------------------------------

I'm not able to do it with "using" clause :(

You should be able to:

using (StreamReader reader = new StreamReader (sourceFile))
{
using (StreamWriter writer = new StreamWriter (tempTarget))
{
string line;
while ( (line=reader.Re adLine()) != null)
{
writer.WriteLin e (line.Replace(t xt1.Text, txt2.text);
}
}
}
File.Delete(sou rceFile);
File.Move(tempT arget,sourceFil e);

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too