473,549 Members | 2,616 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Fast way to read a text file line by line

Hi,
currently I am reading a huge (about 10-100 MB) text-file line by line
using
fstreams and getline. I wonder whether there is a faster way to read a
file line by line (with std::string line). Is there some way to burst
read the whole file and later "extract" each line?

Thanks in advance,
Thomas Kowalski

May 10 '07 #1
6 32847
On 5月10日, 下午7时49分, Thomas Kowalski <t...@gmx.dewro te:
Hi,
currently I am reading a huge (about 10-100 MB) text-file line by line
using
fstreams and getline. I wonder whether there is a faster way to read a
file line by line (with std::string line). Is there some way to burst
read the whole file and later "extract" each line?

Thanks in advance,
Thomas Kowalski
You can open your file with binary mode, and use
ifstream::read( char_type* s, streamsize n) read whole file data into a
large memory buffer in a stringstream object. Then use
stringstream::g etline() to read the data line by line.

May 10 '07 #2
On May 10, 7:49 am, Thomas Kowalski <t...@gmx.dewro te:
Hi,
currently I am reading a huge (about 10-100 MB) text-file line by line
using
fstreams and getline. I wonder whether there is a faster way to read a
file line by line (with std::string line). Is there some way to burst
read the whole file and later "extract" each line?
If performance really is a problem there are a number of things that
can be done. However, most are outside the realm of C++.

Check to see if your string class deallocates when it gets a smaller
string. If not (likey) preallocate your string to a size larger than
what you expect the longest record to be and be sure you use the same
string object to read. You don't want repeated allocations and
deallocations within your string.

You can always use the read() function to get data and parse through
it.

On some systems (e.g. Windoze and VMS) you can map your file to memory.

May 10 '07 #3
<fa**********@y ahoo.comwrote:
On May 10, 7:49 am, Thomas Kowalski <t...@gmx.dewro te:
>Hi,
currently I am reading a huge (about 10-100 MB) text-file line by line
using
fstreams and getline. I wonder whether there is a faster way to read a
file line by line (with std::string line). Is there some way to burst
read the whole file and later "extract" each line?

If performance really is a problem there are a number of things that
can be done. However, most are outside the realm of C++.

Check to see if your string class deallocates when it gets a smaller
string. If not (likey) preallocate your string to a size larger than
what you expect the longest record to be and be sure you use the same
string object to read. You don't want repeated allocations and
deallocations within your string.

You can always use the read() function to get data and parse through
it.
read() is going to operate on arbitray sized "chunks" Be careful to be sure
that *someone* is handdling the seam problems. Somehow detect and handle
the fact that the last fragment of bytes in a chunk is probaly not a a full
and complete line.
May 10 '07 #4
You can open your file with binary mode, and use
ifstream::read( char_type* s, streamsize n) read whole file data into a
large memory buffer in a stringstream object.
Sounds good, but how exactly to I use read something to stringstreams
buffer using ifstream? Somehow I have a blank there.

Thanks in advance,
Thomas Kowalski


May 18 '07 #5
On May 10, 1:49 pm, Thomas Kowalski <t...@gmx.dewro te:
currently I am reading a huge (about 10-100 MB) text-file line
by line using fstreams and getline. I wonder whether there is
a faster way to read a file line by line (with std::string
line). Is there some way to burst read the whole file and
later "extract" each line?
The fastest solution is probably mmap, or it's equivalent under
Windows, but that's very system dependent. Other than that, you
can read the file in one go using something like:
std::istringstr eam tmp ;
tmp << file.rdbuf() ;
std::string s = tmp.str() ;
or:
std::string s( (std::istreambu f_iterator< char >( file )),
(std::istreambu f_iterator< char >()) ) ;

If you can get a good estimation of the size of the file before
hand (which again requires sytem dependent code), then using
reserve on the string in the second example above could
significantly improve performance; as might something like:

std::string s ;
s.resize( knownFileSize ) ;
file.read( &s[ 0 ], s.size() ) ;

The only system I know where it is even possible to get the
exact size of a text file is Unix, however; under Windows, all
of the techniques overstate the size somewhat (and under other
systems, it might not even be possible to get a reasonable
estimate). So you might want to do something like:
s.resize( file.gcount() ) ;
after the above. (It's actually a little bit more complicated.
If there are not at least s.size() bytes in the file---and under
Windows, this will usually be the case if the file is opened as
text, and GetFileSizeEx was used to obtain the size---then
file.read, above, will appear to fail. In fact, if gcount() is
greater than 0, it will have successfully read gcount() bytes,
and if eof() has been set, the read can be considered to have
successfully read all of the bytes in the file.)

(Note that this is not guaranteed under the current C++
standard. It works in practice, however, on all existing
implementations of the library, and will be guaranteed in the
next version of the standard.)

Two other things that you might try:

-- using std::vector< char instead of std::string---with some
implementations , it can be faster (especially if you
construct the string using istreambuf_iter ator), and

-- reading the file as binary, rather than text, and handling
the different end of line representations manually in your
own code.

Concerning the latter, be aware that on some systems, you cannot
open a file as binary if it was created as text, and vice versa.
Just ignoring extra '\r' in the text is often sufficient,
however, for this solution to work adequately under both Unix
and Windows; ignoring only the '\r' which immediately precede a
'\n' is even more correct, but often not worth the extra bother.
And if the file is read as binary, both stat (under Unix) and
GetFileSizeEx (under Windows) will return the exact number of
bytes you can read from it.

--
James Kanze (GABI Software) email:ja******* **@gmail.com
Conseils en informatique orient閑 objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place S閙ard, 78210 St.-Cyr-l'蒫ole, France, +33 (0)1 30 23 00 34

May 18 '07 #6
On May 10, 1:49 pm, Thomas Kowalski <t...@gmx.dewro te:
Hi,
currently I am reading a huge (about 10-100 MB) text-file line by line
using
fstreams and getline. I wonder whether there is a faster way to read a
file line by line (with std::string line). Is there some way to burst
read the whole file and later "extract" each line?
Yes, just write streambuf class that will do that for you.
I did same thing, but not because of performance reasons
but support for large files on 32 bit implementation of streams.

What I did is something like this :

class ImportInBuf: public std::streambuf {
public:
ImportInBuf(con st char* filename)
:fd_(open(filen ame,O_RDONLY | O_LARGEFILE))
{
if(fd_<0)throw std::runtime_er ror(strerror(er rno));
setg(buffer_,bu ffer_,buffer_);
struct stat st;
fstat(fd_,&st);
fsize_=st.st_si ze;
}
// you don;t have to do it like this if your streams are 64 bit
void seekg(uint64_t pos)
{
lseek(fd_,pos,S EEK_SET);
pos_=pos;
setg(buffer_,bu ffer_,buffer_);
}
uint64_t tellg()const { return pos_; }
uint64_t size()const { return fsize_; }
~ImportInBuf(){ close(fd_); }
private:
ImportInBuf(con st ImportInBuf&);
ImportInBuf& operator=(const ImportInBuf&);
virtual int underflow()
{
if(gptr()<egptr ())
{
return *gptr();
}
int size = read(fd_,buffer _,4096);
if(size)
{
pos_+=size;
setg(buffer_,bu ffer_,buffer_+s ize);
return *gptr();
}
pos_=fsize_;
return EOF;
}
char buffer_[4096];
int fd_;
uint64_t fsize_,pos_;
};

Use it like this:
ImportInBuf buf("sample.txt ");
std::istream is(&buf);

or derive class from istream and pass buf in initializer list.
Greetings, Branimir.

May 18 '07 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
5898
by: rishka | last post by:
Rishka Mar 17, 5:40 am show options Newsgroups: comp.databases.oracle.tools From: "Rishka" <ris...@webmail.co.za> - Find messages by this author Date: 17 Mar 2005 05:40:45 -0800 Local: Thurs, Mar 17 2005 5:40 am Subject: Read Text file into Oracle Reports Reply | Reply to Author | Forward | Print | Individual Message | Show original...
5
7619
by: Mika M | last post by:
Hi! I'm trying to read text file like... "Field1";"Field2";"Field3";"Field4" "ABCD";"EFGH";"1234";"20051020" "AABB";"CCDD";"2468";"20051021" "CCDD";"XXYY";"4321";"20051022" ....using OLE DB-provider. Unfortunately I can't affect in what kind of
1
2325
by: sallu2000 | last post by:
hi to all, I am new to php,can u give any idea to read text file from folder based on datewise .for eg. in that folder there are three text file. one.txt 10/3/2007 two.txt 13/3/2007 three.txt 13/3/2007
1
2668
by: abtet | last post by:
Hello people, I am new to php and have question on how to read text file and convert it to XML with php. . Can it be done if the text file contains text, image and also tables? please help and if anyone has the sources that could help solve my problem, please direct me to the source. Thank you very much -abtet-
1
8780
by: neveen | last post by:
i want to open and read text file using j2me that can run on mobile 6630 then i want to make button called read that when pressed the data inside text display
6
2468
by: portCo | last post by:
Hello there, I am creating a vb application which is some like like a questionare. Application read a text file which contains many questions and display one question and the input is needed from user to calculate the score. Here is a problem. I can read a text file. However, it's read whole file at a time. So,
9
2971
by: =?Utf-8?B?QnJpYW4gQ29vaw==?= | last post by:
I want to open a text file and format it into a specific line and then apply color to a specific location of the text and then display it in a RichTextBox after all of this is done. I can do all of the above after the file is loaded into the RichTextBox, and I am trying to speed the process up by doing it in a temp file.
1
1584
aryanraj
by: aryanraj | last post by:
Aryan Hi Everybody, I want to read the text file contents line by line and place the contents into a textarea, using the java script, is there any way to do it. Thanks in Advance, Aryan.
28
3181
by: tlpell | last post by:
Hey, read some tips/pointers on PHP.net but can't seem to solve this problem. I have a php page that reads the contents of a file and then displays the last XX lines of the file. Problem is this...whenever the file gets larger that ~5MB, the page just displays nothing, as though a timeout has occurred but I get no error. At 4.8MB (last...
8
12383
by: karimkhan | last post by:
I am trying to read text file , here is the code , I cant use activex control so plz give me some way.... <html> <head> <script type="text/javascript"> function r() {
0
7730
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7975
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
0
7823
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6059
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project梡lanning, coding, testing, and deployment梬ithout human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5381
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
5101
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
1
1957
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
1069
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
0
777
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.