473,394 Members | 1,382 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,394 software developers and data experts.

Reading heavy .TXT file with StreamReader

I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog):

---------------------------------
System.IO.StreamReader sr = new System.IO.StreamReader(@"C:\1.txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace(",",".");
System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\1.txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object..........

Any help will be greatly appreciated.

Nov 17 '05 #1
13 3927
<ml*******@gmail.com> wrote:
[replacing characters in a 500 MB text file]
[...]
string all = sr.ReadToEnd();


This means you're reading the entire file into memory at once.
Instead, try having a StreamReader and a StreamWriter open at the same
time. Read a line from one file and write it to the other file, then
delete the first file when done and replace it with the second one.
You will only use the memory taken up by one line, rather than
slurping up 500 MB out of nowhere.

P.
Nov 17 '05 #2
Hi,
Below is a better solution:

string sourceFile = @"C:\1.txt";
string tempTarget = System.IO.Path.GetTempFileName();

StreamReader reader = new System.IO.StreamReader( sourceFile );
StreamWriter writer = new StreamWriter( tempTarget )

string line;
while ( (line = reader.ReadLine() )
writer.WriteLine( line.Replace( ",",".") )
reader.Close();
writer.Close();
File.Delete( source);
File.Copy( target, source);
cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
<ml*******@gmail.com> wrote in message
news:11**********************@f14g2000cwb.googlegr oups.com...
I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog):

---------------------------------
System.IO.StreamReader sr = new System.IO.StreamReader(@"C:\1.txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace(",",".");
System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\1.txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object..........

Any help will be greatly appreciated.

Nov 17 '05 #3
He probably wants to do a File.Move instead of a File.Copy for that last
one.

"Ignacio Machin ( .NET/ C# MVP )" <ignacio.machin AT dot.state.fl.us> wrote
in message news:Og****************@TK2MSFTNGP12.phx.gbl...
Hi,
Below is a better solution:

string sourceFile = @"C:\1.txt";
string tempTarget = System.IO.Path.GetTempFileName();

StreamReader reader = new System.IO.StreamReader( sourceFile );
StreamWriter writer = new StreamWriter( tempTarget )

string line;
while ( (line = reader.ReadLine() )
writer.WriteLine( line.Replace( ",",".") )
reader.Close();
writer.Close();
File.Delete( source);
File.Copy( target, source);
cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
<ml*******@gmail.com> wrote in message
news:11**********************@f14g2000cwb.googlegr oups.com...
I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog):

---------------------------------
System.IO.StreamReader sr = new System.IO.StreamReader(@"C:\1.txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace(",",".");
System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\1.txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object..........

Any help will be greatly appreciated.


Nov 17 '05 #4
Hi,

Yes, you are right :)

cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Pete Davis" <pdavis68@[nospam]hotmail.com> wrote in message
news:2I********************@giganews.com...
He probably wants to do a File.Move instead of a File.Copy for that last
one.

"Ignacio Machin ( .NET/ C# MVP )" <ignacio.machin AT dot.state.fl.us>
wrote in message news:Og****************@TK2MSFTNGP12.phx.gbl...
Hi,
Below is a better solution:

string sourceFile = @"C:\1.txt";
string tempTarget = System.IO.Path.GetTempFileName();

StreamReader reader = new System.IO.StreamReader( sourceFile );
StreamWriter writer = new StreamWriter( tempTarget )

string line;
while ( (line = reader.ReadLine() )
writer.WriteLine( line.Replace( ",",".") )
reader.Close();
writer.Close();
File.Delete( source);
File.Copy( target, source);
cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation
<ml*******@gmail.com> wrote in message
news:11**********************@f14g2000cwb.googlegr oups.com...
I must read a very heavy-weight text plain file (usually .txt
extension) )and replace a given character with another given character
in all text inside the file. My application was working pretty well
with this below shown code (code placed in a buttonclick event after
selecting the file in a normal OpenFileDialog):

---------------------------------
System.IO.StreamReader sr = new System.IO.StreamReader(@"C:\1.txt");
string all = sr.ReadToEnd();
sr.Close();
all = all.Replace(",",".");
System.IO.StreamWriter sw = new System.IO.StreamWriter(@"C:\1.txt",
false);
sw.Write(all);
sw.Flush();
sw.Close();
MessageBox.Show("Done!");
---------------------------------

The problem is that user now needs to read over 500MBytes .txt files.
Incredible and hellish but true!

User now runs the application and after selecting .txt file with
OpenFileDialog, application starts processing but inmediatelly crashes
or hangs, I don't remember.

I need a good performance way for reading so heavy .txt files. Reading
the file in different steps would do it? I mean, reading in arrays of
bytes or similar, with MemoryStream object..........

Any help will be greatly appreciated.



Nov 17 '05 #5
Thanks guys, specially Ignacio.

I'll try next days but really seems to be a good solution. If I have
any more problems, I'll return to the post.

Thanks again.

Nov 17 '05 #6
Hi again, the "while" statement does not compile because the while
comparison must return true and we are doing an assigment statement.
ReadLine() returns string and must be bool. Besides, I cannot do
while (reader.ReadLine() != String.Empty) because the file can contain
blank lines and this way we will go out of the while loop.

Any other idea?

Thanks.

Nov 17 '05 #7
It should be
while ( (line.reader.ReadLine()) != null)

Another suggestion, however - rather than having manual calls to Close,
you should use "using" statements round both the reader and the writer.
That way they automatically get closed whether an exception occurs or
not.

Jon

Nov 17 '05 #8
Hi guys,

The correct code, perfectly working, although I haven't tried with 500
MBytes .txt files:

-----------------------------------------------------------------------
string sourceFile = textBox1.Text;
string tempTarget = System.IO.Path.GetTempFileName();
StreamReader reader = new System.IO.StreamReader(sourceFile);
StreamWriter writer = new StreamWriter(tempTarget);

string line = String.Empty;
while ( (line = reader.ReadLine()) != null)
{
writer.WriteLine(line.Replace(txt1.Text,txt2.Text) );
}
reader.Close();
writer.Close();
File.Delete(sourceFile);
File.Move(tempTarget,sourceFile);
----------------------------------------------------------------------------

I'm not able to do it with "using" clause :(

Thanks Jon.

Nov 17 '05 #9
<ml*******@gmail.com> wrote:
Hi guys,

The correct code, perfectly working, although I haven't tried with 500
MBytes .txt files:

-----------------------------------------------------------------------
string sourceFile = textBox1.Text;
string tempTarget = System.IO.Path.GetTempFileName();
StreamReader reader = new System.IO.StreamReader(sourceFile);
StreamWriter writer = new StreamWriter(tempTarget);

string line = String.Empty;
while ( (line = reader.ReadLine()) != null)
{
writer.WriteLine(line.Replace(txt1.Text,txt2.Text) );
}
reader.Close();
writer.Close();
File.Delete(sourceFile);
File.Move(tempTarget,sourceFile);
----------------------------------------------------------------------------

I'm not able to do it with "using" clause :(


You should be able to:

using (StreamReader reader = new StreamReader (sourceFile))
{
using (StreamWriter writer = new StreamWriter (tempTarget))
{
string line;
while ( (line=reader.ReadLine()) != null)
{
writer.WriteLine (line.Replace(txt1.Text, txt2.text);
}
}
}
File.Delete(sourceFile);
File.Move(tempTarget,sourceFile);

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Nov 17 '05 #10
Congratulations for this simple but really good snipplet of code Jon.

Nested "Using" structure seems to be very comfortable and as you said,
you don't have to worry about closing neither the reader nor the
writer.

Thanks very much.

Regards,

Nov 17 '05 #11
Sorry to dissapoint you but my programming style goes more towards
Jon's style. Code looks like much more estructured and clearer for me.
I think in the example you only are "saving" two brackets, don't you?

One of my own programming rules: "It's better to have a clear and
well-structured code (brackets and so on...) in spite of having a
little bit more lines of code".

Thanks anyway. I didn't know you couldn't nest two "using" loops
without any brackets between them.

Regards.

Nov 17 '05 #12
Lonifasiko <ml*******@gmail.com> wrote:
Sorry to dissapoint you but my programming style goes more towards
Jon's style. Code looks like much more estructured and clearer for me.
I think in the example you only are "saving" two brackets, don't you?


It depends. You can easily get really heavily indented code with no
benefit in some situations - it's quite easy to end up with four nested
brackets without much in between.

I very rarely use the version with no extra brackets, but occasionally
it does improve readability. After all, you're treating the whole
contents as a single block.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Nov 17 '05 #13
> Sorry to dissapoint you but my programming style goes more towards
Jon's style. Code looks like much more estructured and clearer for me.
I think in the example you only are "saving" two brackets, don't you?
Thats ok, no dissapointment. What ever make you more productive.
One of my own programming rules: "It's better to have a clear and
well-structured code (brackets and so on...) in spite of having a
little bit more lines of code".


I would disagree with that. It is more clear, IMO, to do it the other way
as you see right away where they come in scope and leave scope at the same
place. For two streams it probably makes no difference. But for 3 or 4 (or
more), using the method I showed can really make it cleaner IMO. Your not
just saving the brackets, but your eliminating all the unnessasary nesting
and indents which can get distracting. And if your stacking streams in a
pipeline (i.e. file to compression to encryption to networkstream), it
really looks good. The end result is the same, but naturally everyone has
different tastes.

--
William Stacey [MVP]

Nov 17 '05 #14

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: jamait | last post by:
Hi all, I m trying to read in a text file into a datatable... Not sure on how to split up the information though, regex or substrings...? sample: Col1 Col2 ...
6
by: Neil Patel | last post by:
I have a log file that puts the most recent record at the bottom of the file. Each line is delimited by a \r\n Does anyone know how to seek to the end of the file and start reading backwards?
2
by: Sabin Finateanu | last post by:
Hi I'm having problem reading a file from my program and I think it's from a procedure I'm using but I don't see where I'm going wrong. Here is the code: public bool AllowUsage() { ...
9
by: haibhoang | last post by:
I have a Windows Service that is trying to parse a large (> 1Gig) text file. I am keep getting OutOfMemoryException exception. Here is the code that's having problem: using (StreamReader...
1
by: Gustaf | last post by:
I'm converting a program I made in Python once to C#, and while I'm at it, I want to do some performance improvements. The program takes a set of mailbox files (in readable formats, like Eudora and...
3
by: leticia larrosa | last post by:
Hi, I try to read a file that have 8 bit character, but contain some character whose code is more than 128 (spanish character). When I read this file using ASCII (Dim oRead As StreamReader =...
9
by: jeff M via .NET 247 | last post by:
I'm still having problems reading EBCDIC files. Currently itlooks like the lower range (0 to 127) is working. I have triedthe following code pages 20284, 20924, 1140, 37, 500 and 20127.By working I...
7
by: Drew Berkemeyer | last post by:
Hello, I'm using the following code to read a text file in VB.NET. Dim sr As StreamReader = File.OpenText(strFilePath) Dim input As String = sr.ReadLine() While Not input Is Nothing...
2
by: James Wong | last post by:
Dear all, I'm using StreamReader to read a text file containing BIG-5 data and found that no matter which encoding method in StreamReader's construction parameter, the BIG-5 contents become...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.