getline buffering

[toton]

Hi,
I am reading some large text files and parsing it. typical file size
I am using is 3 MB. It takes around 20 sec just to use std::getline (I
need to treat newlines properly ) for whole file in debug , and 8 sec
while optimization on.
It is for Visual Studio 7.1 and its std library. While vim opens it
in a fraction of sec.
So, is it that getline is reading the file line by line, instead
reading a chunk at a time in its internal buffer? is there any
function to set how much to read from the stream internally ?
I am not very comfortable with read and readsome , to load a large
buffer, as it changes the file position. While I need the visible file
position to be the position I am actually, while "internally" it
should read some more , may be like 1MB chunk ... ?

abir

[=?iso-8859-1?q?Erik_Wikstr=F6m?=]

On Feb 19, 12:44 pm, "toton" <[e-mail]:

Hi,
I am reading some large text files and parsing it. typical file size
I am using is 3 MB. It takes around 20 sec just to use std::getline (I
need to treat newlines properly ) for whole file in debug , and 8 sec
while optimization on.
It is for Visual Studio 7.1 and its std library. While vim opens it
in a fraction of sec.
So, is it that getline is reading the file line by line, instead
reading a chunk at a time in its internal buffer? is there any
function to set how much to read from the stream internally ?
I am not very comfortable with read and readsome , to load a large
buffer, as it changes the file position. While I need the visible file
position to be the position I am actually, while "internally" it
should read some more , may be like 1MB chunk ... ?

I'm not sure, but I think it's the other way around, Vim does not read
the whole file at once so it's faster.

Each ifstream has a buffer associated with it, you can get a pointer
to it with the rdbuf()-method and you can specify an array to use as
buffer with the pubsetbuf()-method. See the following link for a short
example: http://www.cplusplus.com/reference/i...pubsetbuf.html

--
Erik Wikström

[toton]

On Feb 19, 5:44 pm, "Erik Wikström" <[e-mail]>
wrote:

On Feb 19, 12:44 pm, "toton" <[e-mail]:

Hi,
I am reading some large text files and parsing it. typical file size
I am using is 3 MB. It takes around 20 sec just to use std::getline (I
need to treat newlines properly ) for whole file in debug , and 8 sec
while optimization on.
It is for Visual Studio 7.1 and its std library. While vim opens it
in a fraction of sec.
So, is it that getline is reading the file line by line, instead
reading a chunk at a time in its internal buffer? is there any
function to set how much to read from the stream internally ?
I am not very comfortable with read and readsome , to load a large
buffer, as it changes the file position. While I need the visible file
position to be the position I am actually, while "internally" it
should read some more , may be like 1MB chunk ... ?

I'm not sure, but I think it's the other way around, Vim does not read
the whole file at once so it's faster.

Each ifstream has a buffer associated with it, you can get a pointer
to it with the rdbuf()-method and you can specify an array to use as
buffer with the pubsetbuf()-method. See the following link for a short
example:http://www.cplusplus.com/reference/i...pubsetbuf.html

--
Erik Wikström

Hi,
I had checked it in a separate console project (multi threaded ) it
is running perfectly, and reads within .8 sec. However the same code
takes 12 sec when running inside my Qt app.
I fear Qt lib is interacting with c++ runtime is some way to cause the
problem ....
May be I need to build the Qt lib a fresh to check what is wrong.
Thanks for answering the question ....

abir

[Jacek Dziedzic]

toton wrote:

On Feb 19, 5:44 pm, "Erik Wikström" <[e-mail]>
wrote:

>On Feb 19, 12:44 pm, "toton" <[e-mail]:

>>Hi,
I am reading some large text files and parsing it. typical file size
I am using is 3 MB. It takes around 20 sec just to use std::getline (I
need to treat newlines properly ) for whole file in debug , and 8 sec
while optimization on.
It is for Visual Studio 7.1 and its std library. While vim opens it
in a fraction of sec.
So, is it that getline is reading the file line by line, instead
reading a chunk at a time in its internal buffer? is there any
function to set how much to read from the stream internally ?
I am not very comfortable with read and readsome , to load a large
buffer, as it changes the file position. While I need the visible file
position to be the position I am actually, while "internally" it
should read some more , may be like 1MB chunk ... ?

I'm not sure, but I think it's the other way around, Vim does not read
the whole file at once so it's faster.

Each ifstream has a buffer associated with it, you can get a pointer
to it with the rdbuf()-method and you can specify an array to use as
buffer with the pubsetbuf()-method. See the following link for a short
example:http://www.cplusplus.com/reference/i...pubsetbuf.html

--
Erik Wikström

Hi,
I had checked it in a separate console project (multi threaded ) it
is running perfectly, and reads within .8 sec. However the same code
takes 12 sec when running inside my Qt app.
I fear Qt lib is interacting with c++ runtime is some way to cause the
problem ....
May be I need to build the Qt lib a fresh to check what is wrong.
Thanks for answering the question ....

Make sure you decouple stream I/O from stdio, i.e. do
std::ios::sync_with_stdio(false);

HTH,
- J/

[P.J. Plauger]

"Jacek Dziedzic" <ja************************@gmail.comwrote in message
news:e3***************************@news.chello.pl. ..

toton wrote:

>On Feb 19, 5:44 pm, "Erik Wikström" <[e-mail]>
wrote:

>>On Feb 19, 12:44 pm, "toton" <[e-mail]:

Hi,
I am reading some large text files and parsing it. typical file size
I am using is 3 MB. It takes around 20 sec just to use std::getline (I
need to treat newlines properly ) for whole file in debug , and 8 sec
while optimization on.
It is for Visual Studio 7.1 and its std library. While vim opens it
in a fraction of sec.
So, is it that getline is reading the file line by line, instead
reading a chunk at a time in its internal buffer? is there any
function to set how much to read from the stream internally ?
I am not very comfortable with read and readsome , to load a large
buffer, as it changes the file position. While I need the visible file
position to be the position I am actually, while "internally" it
should read some more , may be like 1MB chunk ... ?
I'm not sure, but I think it's the other way around, Vim does not read
the whole file at once so it's faster.

Each ifstream has a buffer associated with it, you can get a pointer
to it with the rdbuf()-method and you can specify an array to use as
buffer with the pubsetbuf()-method. See the following link for a short
example:http://www.cplusplus.com/reference/i...pubsetbuf.html

--
Erik Wikström

Hi,
I had checked it in a separate console project (multi threaded ) it
is running perfectly, and reads within .8 sec. However the same code
takes 12 sec when running inside my Qt app.
I fear Qt lib is interacting with c++ runtime is some way to cause the
problem ....
May be I need to build the Qt lib a fresh to check what is wrong.
Thanks for answering the question ....

Make sure you decouple stream I/O from stdio, i.e. do
std::ios::sync_with_stdio(false);

Normally good advice, but unnecessary with VC++.

P.J. Plauger
Dinkumware, Ltd.
http://www.dinkumware.com

[toton]

On Feb 19, 8:49 pm, "P.J. Plauger" <[e-mail]:

"Jacek Dziedzic" <[e-mail] in message

news:e3***************************@news.chello.pl. ..

toton wrote:

On Feb 19, 5:44 pm, "Erik Wikström" <[e-mail]>
wrote:
On Feb 19, 12:44 pm, "toton" <[e-mail]:

>>Hi,
I am reading some large text files and parsing it. typical file size
I am using is 3 MB. It takes around 20 sec just to use std::getline (I
need to treat newlines properly ) for whole file in debug , and 8 sec
while optimization on.
It is for Visual Studio 7.1 and its std library. While vim opens it
in a fraction of sec.
So, is it that getline is reading the file line by line, instead
reading a chunk at a time in its internal buffer? is there any
function to set how much to read from the stream internally ?
I am not very comfortable with read and readsome , to load a large
buffer, as it changes the file position. While I need the visible file
position to be the position I am actually, while "internally" it
should read some more , may be like 1MB chunk ... ?
I'm not sure, but I think it's the other way around, Vim does not read
the whole file at once so it's faster.

>Each ifstream has a buffer associated with it, you can get a pointer
to it with the rdbuf()-method and you can specify an array to use as
buffer with the pubsetbuf()-method. See the following link for a short
example:http://www.cplusplus.com/reference/i...pubsetbuf.html

>--
Erik Wikström

Hi,
I had checked it in a separate console project (multi threaded ) it
is running perfectly, and reads within .8 sec. However the same code
takes 12 sec when running inside my Qt app.
I fear Qt lib is interacting with c++ runtime is some way to cause the
problem ....
May be I need to build the Qt lib a fresh to check what is wrong.
Thanks for answering the question ....

Make sure you decouple stream I/O from stdio, i.e. do
std::ios::sync_with_stdio(false);

Normally good advice, but unnecessary with VC++.

P.J. Plauger
Dinkumware, Ltd.http://www.dinkumware.com

I got the problem. It has nothing to do with Qt or other
libraries ....
I was using a tellg() to get the current position. Now my question is
why tellg is such costly ? Won't it just return the current strem
position ?
To explain,
{
boost::progress_timer t;
std::ifstream in("Y:/Data/workspaces/tob4f/tob4f.dat");
std::string line;
while(in){
///int pos = in.tellg();
std::getline(in,line);
}
}
This code takes 0.58 sec in my computer while if I uncomment the line
in.tellg(), it takes 120.8 sec !

[Ismo Salonen]

toton wrote:

On Feb 19, 8:49 pm, "P.J. Plauger" <[e-mail]:

>"Jacek Dziedzic" <[e-mail] in message

news:e3***************************@news.chello.pl ...

>>toton wrote:
On Feb 19, 5:44 pm, "Erik Wikström" <[e-mail]>
wrote:
On Feb 19, 12:44 pm, "toton" <[e-mail]:
>Hi,
> I am reading some large text files and parsing it. typical file size
>I am using is 3 MB. It takes around 20 sec just to use std::getline (I
>need to treat newlines properly ) for whole file in debug , and 8 sec
>while optimization on.
> It is for Visual Studio 7.1 and its std library. While vim opens it
>in a fraction of sec.
> So, is it that getline is reading the file line by line, instead
>reading a chunk at a time in its internal buffer? is there any
>function to set how much to read from the stream internally ?
> I am not very comfortable with read and readsome , to load a large
>buffer, as it changes the file position. While I need the visible file
>position to be the position I am actually, while "internally" it
>should read some more , may be like 1MB chunk ... ?
I'm not sure, but I think it's the other way around, Vim does not read
the whole file at once so it's faster.
Each ifstream has a buffer associated with it, you can get a pointer
to it with the rdbuf()-method and you can specify an array to use as
buffer with the pubsetbuf()-method. See the following link for a short
example:http://www.cplusplus.com/reference/i...pubsetbuf.html
--
Erik Wikström
Hi,
I had checked it in a separate console project (multi threaded ) it
is running perfectly, and reads within .8 sec. However the same code
takes 12 sec when running inside my Qt app.
I fear Qt lib is interacting with c++ runtime is some way to cause the
problem ....
May be I need to build the Qt lib a fresh to check what is wrong.
Thanks for answering the question ....
Make sure you decouple stream I/O from stdio, i.e. do
std::ios::sync_with_stdio(false);

Normally good advice, but unnecessary with VC++.

P.J. Plauger
Dinkumware, Ltd.http://www.dinkumware.com

I got the problem. It has nothing to do with Qt or other
libraries ....
I was using a tellg() to get the current position. Now my question is
why tellg is such costly ? Won't it just return the current strem
position ?
To explain,
{
boost::progress_timer t;
std::ifstream in("Y:/Data/workspaces/tob4f/tob4f.dat");
std::string line;
while(in){
///int pos = in.tellg();
std::getline(in,line);
}
}
This code takes 0.58 sec in my computer while if I uncomment the line
in.tellg(), it takes 120.8 sec !

Could it be that you have opened the file in text mode and the tellg()
seeks to beginning always and rereads characters (counting cr+lf pairs
as one ). Try switching to binary mode and handle cr+lf yourself.

ismo

[toton]

On Feb 20, 11:10 am, Ismo Salonen <[e-mail]:

toton wrote:

On Feb 19, 8:49 pm, "P.J. Plauger" <[e-mail]:

"Jacek Dziedzic" <[e-mail] in message

>news:e3***************************@news.chello.pl ...

>toton wrote:
On Feb 19, 5:44 pm, "Erik Wikström" <[e-mail]>
wrote:
On Feb 19, 12:44 pm, "toton" <[e-mail]:
Hi,
I am reading some large text files and parsing it. typical file size
I am using is 3 MB. It takes around 20 sec just to use std::getline (I
need to treat newlines properly ) for whole file in debug , and 8 sec
while optimization on.
It is for Visual Studio 7.1 and its std library. While vim opens it
in a fraction of sec.
So, is it that getline is reading the file line by line, instead
reading a chunk at a time in its internal buffer? is there any
function to set how much to read from the stream internally ?
I am not very comfortable with read and readsome , to load a large
buffer, as it changes the file position. While I need the visible file
position to be the position I am actually, while "internally" it
should read some more , may be like 1MB chunk ... ?
I'm not sure, but I think it's the other way around, Vim does not read
the whole file at once so it's faster.
Each ifstream has a buffer associated with it, you can get a pointer
to it with the rdbuf()-method and you can specify an array to use as
buffer with the pubsetbuf()-method. See the following link for a short
example:http://www.cplusplus.com/reference/i...pubsetbuf.html
--
Erik Wikström
Hi,
I had checked it in a separate console project (multi threaded ) it
is running perfectly, and reads within .8 sec. However the same code
takes 12 sec when running inside my Qt app.
I fear Qt lib is interacting with c++ runtime is some way to cause the
problem ....
May be I need to build the Qt lib a fresh to check what is wrong.
Thanks for answering the question ....
Make sure you decouple stream I/O from stdio, i.e. do
std::ios::sync_with_stdio(false);
Normally good advice, but unnecessary with VC++.

P.J. Plauger
Dinkumware, Ltd.http://www.dinkumware.com

I got the problem. It has nothing to do with Qt or other
libraries ....
I was using a tellg() to get the current position. Now my question is
why tellg is such costly ? Won't it just return the current strem
position ?
To explain,
{
boost::progress_timer t;
std::ifstream in("Y:/Data/workspaces/tob4f/tob4f.dat");
std::string line;
while(in){
///int pos = in.tellg();
std::getline(in,line);
}
}
This code takes 0.58 sec in my computer while if I uncomment the line
in.tellg(), it takes 120.8 sec !

Could it be that you have opened the file in text mode and the tellg()
seeks to beginning always and rereads characters (counting cr+lf pairs
as one ). Try switching to binary mode and handle cr+lf yourself.

ismo

The whole purpose of using getline is that only. I am not sure why
tellg have to behave like that in text mode , it is stored one !
Tested the same with gcc .The program in mingw is not giving any big
performance
difference.
here is the program
#include <fstream>
#include <iostream>
#include <ctime>
int main(){
{
//boost::progress_timer t;
time_t start,end;
time(&start);
std::ifstream in("Y:/Data/workspaces/tob4f/tob4f.dat");
std::string line;
while(in){
int pos = in.tellg();
std::getline(in,line);
}
time(&end);
std::cout<<difftime(end,start);
}
}
With & without comment on the line , it takes 2 sec & 3 sec
respectively (without -o2 flag ) It looks fine to me ...
Even the visual studio std code looks quite simple one ....
anyone else has tested it with a big file (4-8 MB )and found a huge
difference ?

thanks
abir

[=?iso-8859-1?q?Erik_Wikstr=F6m?=]

On Feb 20, 7:45 am, "toton" <[e-mail]:

On Feb 20, 11:10 am, Ismo Salonen <[e-mail]:

toton wrote:

On Feb 19, 8:49 pm, "P.J. Plauger" <[e-mail]:
>"Jacek Dziedzic" <[e-mail] in message

>>news:e3***************************@news.chello.p l...

>>toton wrote:
>>>On Feb 19, 5:44 pm, "Erik Wikström" <[e-mail]>
>>>wrote:
>>>>On Feb 19, 12:44 pm, "toton" <[e-mail]:
>>>>>Hi,
>>>>> I am reading some large text files and parsing it. typical file size
>>>>>I am using is 3 MB. It takes around 20 sec just to use std::getline (I
>>>>>need to treat newlines properly ) for whole file in debug , and 8 sec
>>>>>while optimization on.
>>>>> It is for Visual Studio 7.1 and its std library. While vim opens it
>>>>>in a fraction of sec.
>>>>> So, is it that getline is reading the file line by line, instead
>>>>>reading a chunk at a time in its internal buffer? is there any
>>>>>function to set how much to read from the stream internally ?
>>>>> I am not very comfortable with read and readsome , to load a large
>>>>>buffer, as it changes the file position. While I need the visible file
>>>>>position to be the position I am actually, while "internally" it
>>>>>should read some more , may be like 1MB chunk ... ?
>>>>I'm not sure, but I think it's the other way around, Vim does notread
>>>>the whole file at once so it's faster.
>>>>Each ifstream has a buffer associated with it, you can get a pointer
>>>>to it with the rdbuf()-method and you can specify an array to useas
>>>>buffer with the pubsetbuf()-method. See the following link for a short
>>>>example:http://www.cplusplus.com/reference/i...pubsetbuf.html
>>>>--
>>>>Erik Wikström
>>>Hi,
>>> I had checked it in a separate console project (multi threaded )it
>>>is running perfectly, and reads within .8 sec. However the same code
>>>takes 12 sec when running inside my Qt app.
>>>I fear Qt lib is interacting with c++ runtime is some way to causethe
>>>problem ....
>>>May be I need to build the Qt lib a fresh to check what is wrong.
>>>Thanks for answering the question ....
>> Make sure you decouple stream I/O from stdio, i.e. do
>>std::ios::sync_with_stdio(false);
>Normally good advice, but unnecessary with VC++.

>P.J. Plauger
>Dinkumware, Ltd.http://www.dinkumware.com

I got the problem. It has nothing to do with Qt or other
libraries ....
I was using a tellg() to get the current position. Now my question is
why tellg is such costly ? Won't it just return the current strem
position ?
To explain,
{
boost::progress_timer t;
std::ifstream in("Y:/Data/workspaces/tob4f/tob4f.dat");
std::string line;
while(in){
///int pos = in.tellg();
std::getline(in,line);
}
}
This code takes 0.58 sec in my computer while if I uncomment the line
in.tellg(), it takes 120.8 sec !

Could it be that you have opened the file in text mode and the tellg()
seeks to beginning always and rereads characters (counting cr+lf pairs
as one ). Try switching to binary mode and handle cr+lf yourself.

ismo

The whole purpose of using getline is that only. I am not sure why
tellg have to behave like that in text mode , it is stored one !
Tested the same with gcc .The program in mingw is not giving any big
performance
difference.
here is the program
#include <fstream>
#include <iostream>
#include <ctime>
int main(){
{
//boost::progress_timer t;
time_t start,end;
time(&start);
std::ifstream in("Y:/Data/workspaces/tob4f/tob4f.dat");
std::string line;
while(in){
int pos = in.tellg();
std::getline(in,line);
}
time(&end);
std::cout<<difftime(end,start);
}}

With & without comment on the line , it takes 2 sec & 3 sec
respectively (without -o2 flag ) It looks fine to me ...
Even the visual studio std code looks quite simple one ....
anyone else has tested it with a big file (4-8 MB )and found a huge
difference ?

On a 22.5MB file I get one second running time without tellg, 4
seconds if the file is opened in text mode and 2 seconds if opened in
binary mode. Seems quite reasonable to me.

--
Erik Wikström