473,320 Members | 2,048 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

How to read large text file ?

Hi, All
I have text file ASCII with record length of 388 bytes, no record delimeter
and size of the file is 562477780 bytes and 1449685 records all togeter.
How can i read such file record by record ?

Please, help
Aug 15 '06 #1
3 1805

elena wrote:
Hi, All
I have text file ASCII with record length of 388 bytes, no record delimeter
and size of the file is 562477780 bytes and 1449685 records all togeter.
How can i read such file record by record ?

Please, help
First of all, I wouldn't read this record-by-record. Disk reads are
slow, and so it is always best to minimize the number of reads if
possible. You can do this by reading in chunks... You will want your
chunk size to be a multiple of 388 (your record length). Then you can
process those records before you read your next chunk. So, in psudo
code you would do something like:

while more records
read group of records
process records
end while

Now, in code you would probably want to use the System.IO.StreamReader
class for this - the built in VB.NET file functions are very slow, and
are best avoided when working with larger files. There are examples in
the documentation of this class that show how to read a specific number
of char's at a time, so I'll direct you to the doc's for that :)

You may also want to consider not hardcoding the number of records to
process. That way, you can optimize it for speed and memory usage :)
I would consider puting the group size in an app.config file.

--
Tom Shelton [MVP]

Aug 15 '06 #2
Elena,

You have first to read the file encoding with ASCII (are you sure it is
ASCII, because that 7 bit code is seldom used).

http://msdn.microsoft.com/library/de...ctortopic9.asp

Than you can go through that using the Mid or the Substring where I prefer
the substring

http://msdn.microsoft.com/library/de...ringTopic2.asp

I hope this helps,

Cor
"elena" <el***@discussions.microsoft.comschreef in bericht
news:E0**********************************@microsof t.com...
Hi, All
I have text file ASCII with record length of 388 bytes, no record
delimeter
and size of the file is 562477780 bytes and 1449685 records all togeter.
How can i read such file record by record ?

Please, help

Aug 15 '06 #3
Thank you so much for your input, now i start.
"Cor Ligthert [MVP]" wrote:
Elena,

You have first to read the file encoding with ASCII (are you sure it is
ASCII, because that 7 bit code is seldom used).

http://msdn.microsoft.com/library/de...ctortopic9.asp

Than you can go through that using the Mid or the Substring where I prefer
the substring

http://msdn.microsoft.com/library/de...ringTopic2.asp

I hope this helps,

Cor
"elena" <el***@discussions.microsoft.comschreef in bericht
news:E0**********************************@microsof t.com...
Hi, All
I have text file ASCII with record length of 388 bytes, no record
delimeter
and size of the file is 562477780 bytes and 1449685 records all togeter.
How can i read such file record by record ?

Please, help


Aug 15 '06 #4

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Chuck Amadi | last post by:
Has anyone got a simple python script that will parse a linux mbox and create a large file to view . Cheers Chu
7
by: CJ | last post by:
Hello! I want to read a large text file (100 mb) and from with in the file i want to take out some lines on a spesific kriteria and make new files. eks. inputfile: 123 hello.. 123 more......
9
by: sweety | last post by:
Dear All, How to encrypt a C data file and make binary file and then have to read a bin file at run time and decrypt the file and have to read the data. Any help to achive this pls. Would be...
35
by: RyanS09 | last post by:
Hello- I am trying to write a snippet which will open a text file with an integer on each line. I would like to read the last integer in the file. I am currently using: file = fopen("f.txt",...
2
by: Jarry | last post by:
I am a bit new to VB, and I have set up a program which loads a large text file. The text file (.txt) has a file size of around 6 megabytes. This is the figure I would like to rreduce. I have...
2
by: starffly | last post by:
I want to read a xml file in Unicode, UTF-8 or a native encoding into a wchar_t type string, so i write a routine as follows, however, sometimes a Unicode file including Chinese character cannot...
3
by: utab | last post by:
Dear all, What are the advantages of binary files over text files? I would like to search for a specific value of a variable in an output file, I was doing this lately by the string library...
3
by: =?Utf-8?B?ZGF2aWQ=?= | last post by:
I try to follow Steve's paper to build a database, and store a small text file into SQL Server database and retrieve it later. Only difference between my table and Steve's table is that I use NTEXT...
2
by: Kevin Ar18 | last post by:
I posted this on the forum, but nobody seems to know the solution: http://python-forum.org/py/viewtopic.php?t=5230 I have a zip file that is several GB in size, and one of the files inside of it...
4
by: Keith G Hicks | last post by:
I'm trying to read a text file and alter the contents of specific lines in the file. I know how to use streamreader to read each line of a file. I'm doing that already to get the data into a...
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.