q: fast file IO

[laniik]

Hi. I am trying to read in a large number of floats from an ASCII
file.

I am currently using i/o streams as such:

vector <float> values;
float val;
ifstream in(fn);
in>>height;
in>>width;

for(int i=0;i<height;i++) {
for(int j=0;j<width;j++) {

in>>val;
values.push_back(val);

}
}

where the data is:

Width Height
value value value value value value...
However, this is turning out to be really slow when I read about a
million points. I was wondering if anyone knows how i could read this
data in much faster? I know that it is possible because several
applications do it already Maybe using C FILEs is faster than streams?
Im not sure.

Thanks!

Oliver

[Mike Austin]

laniik wrote:

Hi. I am trying to read in a large number of floats from an ASCII
file.

I am currently using i/o streams as such:

vector <float> values;
float val;
ifstream in(fn);
in>>height;
in>>width;

for(int i=0;i<height;i++) {
for(int j=0;j<width;j++) {

in>>val;
values.push_back(val);

}
}

where the data is:

Width Height
value value value value value value...
However, this is turning out to be really slow when I read about a
million points. I was wondering if anyone knows how i could read this
data in much faster? I know that it is possible because several
applications do it already Maybe using C FILEs is faster than streams?
Im not sure.

Thanks!

Oliver

You could try this:

#include <vector>
#include <fstream>
#include <iterator>

using namespace std;

int main() {
ifstream file( filename );
int width, height;

file >> width >> height;
vector<float> values( width * height );

copy( istream_iterator<float>( file ), istream_iterator<float>(),
back_inserter( values ) );

return 0;
}

This was created with a little help from
http://www.sgi.com/tech/stl/istream_iterator.html.

Mike

[Mike Wahler]

"Mike Austin" <[e-mail]> wrote in message
news:zi********************@bgtnsc04-news.ops.worldnet.att.net...

laniik wrote:

Hi. I am trying to read in a large number of floats from an ASCII
file.

I am currently using i/o streams as such:

vector <float> values;
float val;
ifstream in(fn);
in>>height;
in>>width;

for(int i=0;i<height;i++) {
for(int j=0;j<width;j++) {

in>>val;
values.push_back(val);

}
}

where the data is:

Width Height
value value value value value value...
However, this is turning out to be really slow when I read about a
million points. I was wondering if anyone knows how i could read this
data in much faster? I know that it is possible because several
applications do it already Maybe using C FILEs is faster than streams?
Im not sure.

Thanks!

Oliver

You could try this:

#include <vector>
#include <fstream>
#include <iterator>

using namespace std;

int main() {
ifstream file( filename );
int width, height;

file >> width >> height;
vector<float> values( width * height );

copy( istream_iterator<float>( file ), istream_iterator<float>(),
back_inserter( values ) );

return 0;

This is unlikely to improve performance, since 'istream_iterator'
uses operator>>(), which 'lanik' was already using. 'lanik's idea
of using a FILE* instead of an ifstream might or might not be
faster, the only way to know is to measure. Another possibility
is to use some nonstandard platform-specific method which might
exploit a more intimate connection with the OS/file system.

-Mike

[Zenith]

It is likely that the slowness is mainly caused by the repeatedly
reallocations of your vector object, rather than file operations. You
may try reserve() to pre-allocate the amount of memory expected.

[Alvin]

Zenith wrote:

It is likely that the slowness is mainly caused by the repeatedly
reallocations of your vector object, rather than file operations. You
may try reserve() to pre-allocate the amount of memory expected.

You can also try a std::list. It wont double in size as a vector tends to
do.

Alvin

--
Why must I click 'Start' in order to turn off my computer?

[Mike Austin]

Mike Wahler wrote:

This is unlikely to improve performance, since 'istream_iterator'
uses operator>>(), which 'lanik' was already using. 'lanik's idea
of using a FILE* instead of an ifstream might or might not be
faster, the only way to know is to measure. Another possibility
is to use some nonstandard platform-specific method which might
exploit a more intimate connection with the OS/file system.

-Mike

Let's try it another way then. If each float string is a specific
length, we can take advantage of that:

#include <iostream>
#include <vector>
#include <iterator>
#include <fstream>

using namespace std;

const int readLength = 9;

int main( int argc, char* argv[] ) {
ifstream file( "float_data" );
int width, height, size;
char buffer[256];

file >> width >> height;
size = width * height
vector<float> values( size );

for( int i = 0; i < size && file.good(); ++i ) {
file.get( buffer, readLength );
values[i] = atof( buffer );
}

vector<float>::iterator i;
for( i = values.begin(); i != values.end(); ++i ) {
cout << *i << " " << flush;
}

return 0;
}

Mike

[laniik]

i dont know this for sure, but I have heard that the atof function is
fairly slow. and that i could speed up reading time by writing my own
atof function. so maybe using .get() with my own atof will be faster
than <<.

also, does anyone know how << turns what it reads in into a float?
wonder how fast that part is.

ill try the memory allocation idea, ill try that, that makes sense.

the list idea might speed things up, but I need to have random access
abilities with this array.

thanks everyone!

[Mike Austin]

laniik wrote:

i dont know this for sure, but I have heard that the atof function is
fairly slow. and that i could speed up reading time by writing my own
atof function. so maybe using .get() with my own atof will be faster
than <<.

also, does anyone know how << turns what it reads in into a float?
wonder how fast that part is.

ill try the memory allocation idea, ill try that, that makes sense.

the list idea might speed things up, but I need to have random access
abilities with this array.

thanks everyone!

I've heard things like this often. The next question would be, why
provide library functions if they are useless and slow? One factor is
of course that the library routines are very general. Here's an
implementation I found with google:

http://www.jbox.dk/sanos/source/lib/strtod.c.html

The scaling loop could be eliminated in trade for accuracy:

int scale = 1;

[...]

if (*p == '.') {
p++;
while (isdigit(*p)) {
number = number * 10. + (*p - '0');
p++;
scale *= 10;
}
}

[...]

return number / scale;
Regards,
Mike

[Samee Zahur]

If you have the option of changing the input file format, try storing
binary data. Use something like this:

union{float num;char data[sizeof(num)];};

That way the big killer (converting floats to and from strings) gets
killed!

Samee