473,320 Members | 2,177 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Reading a large text file line by line backwards

Is there a way through .net to read a very large text file (400MB+)
backwards line by line. In system.io the filestream class has a "seek"
method but the only read method requires you to know how many bytes to read
in.

My problem is that the line length of this log file is not constant so there
is no easy way to read one line in. The only thing that is constant is that
each line is terminated by a carriage return.

If my only option was to use the filestream class I suppose I can read data
back in chunks and try to parse the data and determine lines...

Is there a better way?
Amy.
Nov 16 '05 #1
1 4504
"Amy L." <am**@paxemail.com> wrote in message
news:ua**************@TK2MSFTNGP10.phx.gbl...
Is there a way through .net to read a very large text file (400MB+)
backwards line by line. In system.io the filestream class has a "seek"
method but the only read method requires you to know how many bytes to read in.

My problem is that the line length of this log file is not constant so there is no easy way to read one line in. The only thing that is constant is that each line is terminated by a carriage return.

If my only option was to use the filestream class I suppose I can read data back in chunks and try to parse the data and determine lines...

Is there a better way?
Amy.


About all you can do is seek to, say, 1024 bytes before the end; read 1024
bytes; work through them byte by byte to chop up into lines; seek to 1024
bytes before that; read 1024 bytes; and keep going.

The problem is that, as you know, "lines" mean nothing to the low-level disk
functions in the operating system. This isn't like an IBM mainframe where
files have a "record length" and are divided into "card images".

I suggested 1024 because the cluster size (allocation unit) is probably a
multiple of 1024, so you will be synchronized with the disk sector
boundaries; that should be mildly advantageous as regards speed.
Nov 16 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

7
by: Jay | last post by:
I have a very large text file (being read by a CGI script on a web server), and I get memory errors when I try to read the whole file into a list of strings. The problem is, I want to read the file...
2
by: Amy L. | last post by:
Is there a way through .net to read a very large text file (400MB+) backwards line by line. In system.io the filestream class has a "seek" method but the only read method requires you to know how...
11
by: mkarja | last post by:
Hi, I'm trying to figure out how to read some range of rows from a file. Is it possible to search the file with some criteria and then when the search string is found read 3 rows before and...
20
by: sahukar praveen | last post by:
Hello, I have a question. I try to print a ascii file in reverse order( bottom-top). Here is the logic. 1. Go to the botton of the file fseek(). move one character back to avoid the EOF. 2....
11
by: Matt DeFoor | last post by:
I have some log files that I'm working with that look like this: 1000000000 3456 1234 1000000001 3456 1235 1000020002 3456 1223 1000203044 3456 986 etc. I'm trying to read the file...
6
by: Rajorshi Biswas | last post by:
Hi folks, Suppose I have a large (1 GB) text file which I want to read in reverse. The number of characters I want to read at a time is insignificant. I'm confused as to how best to do it. Upon...
6
by: Neil Patel | last post by:
I have a log file that puts the most recent record at the bottom of the file. Each line is delimited by a \r\n Does anyone know how to seek to the end of the file and start reading backwards?
3
by: booksnore | last post by:
I have to read data from a flat file with millions of records. I wanted to find the most efficient way of doing this. I was just going to use a StreamReader and then break up the input line using...
1
by: stoogots2 | last post by:
I have written a Windows App in C# that needs to read a text file over the network, starting from the end of the file and reading backwards toward the beginning (looking for the last occurrence of a...
3
by: xyz | last post by:
Hi, I have a text file around 7GB includes 100 million lines... I want to read the data line by line when I approach my module.. ie., when i read for the first time , my program shuld read only...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.