473,386 Members | 1,819 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,386 software developers and data experts.

Reading and Processing Text File Efficiently

Hi,

I am writing a program that reads a text file and populate its contents into
a DBGrid
Each row inside is considered an record.
Each record is comprised of several fixed length columns
i.e: Product Column - 9, Barcode - 13, Width - 9, Weight - 9
Ex:
Prod-001 001-200-300 550.00 1100.2567
Prod-002 002-200-400 7100.00 1100.2567
Prod-0A 001-210-300 71000.00 1100.2567

I have 2 questions:-
1) Since the length/size of the text file can be vary.
How to write a code to get the no of lines in the files without looping
through the whole file ?

2) Which is more efficient ? Use collection or Dataset/Datatable to hold
those data?
Thank You.

Nov 21 '05 #1
4 1226
"Woo Mun Foong" <mf***@yahoo.com> wrote in message
news:BD**********************************@microsof t.com...
Hi,

I am writing a program that reads a text file and populate its contents
into
a DBGrid
Each row inside is considered an record.
Each record is comprised of several fixed length columns
i.e: Product Column - 9, Barcode - 13, Width - 9, Weight - 9
Ex:
Prod-001 001-200-300 550.00 1100.2567
Prod-002 002-200-400 7100.00 1100.2567
Prod-0A 001-210-300 71000.00 1100.2567

I have 2 questions:-
1) Since the length/size of the text file can be vary.
How to write a code to get the no of lines in the files without looping
through the whole file ?

2) Which is more efficient ? Use collection or Dataset/Datatable to hold
those data?
Thank You.


In answer to your first question, if the length of each line is variable
then there isn't any way to get the number of lines without reading through
the file. If you are worried about performance then think about caching the
result somewhere.

Sorry, can't advise on your second question.

Brian.

--
_ _ _ __ ___ ____ ______ ____ ___ __ _ _ _ _

Brian Cryer
www.cryer.co.uk/brian
_ _ _ __ ___ ____ ______ ____ ___ __ _ _ _ _


Nov 21 '05 #2

My way of doing this was to read the file into an array. Mine had a little
more data to shuffle through but read it to the array very quickly. Once in
the array the user was notified of the number of records and some starting
and ending data. Then the user could choose at what record to start the
import and when.

Would be happy to send some of the code if this is what you wish to do.

Thomas

--
Posted via NewsDemon.com - Premium Uncensored Newsgroup Service
------->>>>>>http://www.NewsDemon.com<<<<<<------
Unlimited Access, Anonymous Accounts, Uncensored Broadband Access
Nov 21 '05 #3
Dear Thomas,

May I ?
I'm sure like to get hold of a piece of your code.

Thank You
mfwoo
"th*****@msala.net" wrote:

My way of doing this was to read the file into an array. Mine had a little
more data to shuffle through but read it to the array very quickly. Once in
the array the user was notified of the number of records and some starting
and ending data. Then the user could choose at what record to start the
import and when.

Would be happy to send some of the code if this is what you wish to do.

Thomas

--
Posted via NewsDemon.com - Premium Uncensored Newsgroup Service
------->>>>>>http://www.NewsDemon.com<<<<<<------
Unlimited Access, Anonymous Accounts, Uncensored Broadband Access

Nov 21 '05 #4
Dear Thomas,

May I ?
I'm sure like to get hold of a piece of your code.

Thank You
mfwoo
"th*****@msala.net" wrote:

My way of doing this was to read the file into an array. Mine had a little
more data to shuffle through but read it to the array very quickly. Once in
the array the user was notified of the number of records and some starting
and ending data. Then the user could choose at what record to start the
import and when.

Would be happy to send some of the code if this is what you wish to do.

Thomas

--
Posted via NewsDemon.com - Premium Uncensored Newsgroup Service
------->>>>>>http://www.NewsDemon.com<<<<<<------
Unlimited Access, Anonymous Accounts, Uncensored Broadband Access

Nov 21 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Kevin T. Ryan | last post by:
Hi All - I'm not sure, but I'm wondering if this is a bug, or maybe (more likely) I'm misunderstanding something...see below: >>> f = open('testfile', 'w') >>> f.write('kevin\n') >>>...
4
by: Xah Lee | last post by:
# -*- coding: utf-8 -*- # Python # to open a file and write to file # do f=open('xfile.txt','w') # this creates a file "object" and name it f. # the second argument of open can be
3
by: Markus Hofmann | last post by:
Hi @ all hope someone can help me as my PC is ready to go through the window...here it is: I have a text file with approximately 150,000 lines of the following format: ...
24
by: Hendrik Schober | last post by:
Hi, I have a 'std::istream' and need to read its whole contents into a string. How can I do this? TIA; Schobi
0
by: Einar Høst | last post by:
Hi, I'm reading data from a text file, and having read a bit on performance and memory, I'm wondering how it could be done in the fastest, most frugal way. The data comes in messages, one per...
4
by: Jason Kumpf | last post by:
OK I've been staring at this code all day and still with everything I have tried I cannot figure out two problems I am having. Once is why the space limit for the directory I create in the code...
4
by: jesuraj | last post by:
Hi, how can i read input from a data file which contains binary or hex values. I have to use the exact binary data for further processing.only limited number of bits are taken form the file(64...
7
by: random guy | last post by:
Hi, I'm writing a program which creates an index of text files. For each file it processes, the program records the start and end positions (as returned by tellg()) of sections of interest,...
16
by: malibuster | last post by:
I have a text source file of about 20.000 lines. that file, grab the next 5 lines write these to a new file... grabbing 5 lines and creating new files until processing of all 20.000 lines is done....
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.