q: fast file IO

Hi. I am trying to read in a large number of floats from an ASCII
file.

I am currently using i/o streams as such:

vector <float> values;
float val;
ifstream in(fn);
in>>height;
in>>width;

for(int i=0;i<height;i++) {
for(int j=0;j<width;j++) {

in>>val;
values.push_back(val);

}
}

where the data is:

Width Height
value value value value value value...


However, this is turning out to be really slow when I read about a
million points. I was wondering if anyone knows how i could read this
data in much faster? I know that it is possible because several
applications do it already Maybe using C FILEs is faster than streams?
Im not sure.

Thanks!

Oliver

re: q: fast file IO

laniik wrote:[color=blue]
> Hi. I am trying to read in a large number of floats from an ASCII
> file.
>
> I am currently using i/o streams as such:
>
> vector <float> values;
> float val;
> ifstream in(fn);
> in>>height;
> in>>width;
>
> for(int i=0;i<height;i++) {
> for(int j=0;j<width;j++) {
>
> in>>val;
> values.push_back(val);
>
> }
> }
>
> where the data is:
>
> Width Height
> value value value value value value...
>
>
> However, this is turning out to be really slow when I read about a
> million points. I was wondering if anyone knows how i could read this
> data in much faster? I know that it is possible because several
> applications do it already Maybe using C FILEs is faster than streams?
> Im not sure.
>
> Thanks!
>
> Oliver[/color]

You could try this:

#include <vector>
#include <fstream>
#include <iterator>

using namespace std;

int main() {
ifstream file( filename );
int width, height;

file >> width >> height;
vector<float> values( width * height );

copy( istream_iterator<float>( file ), istream_iterator<float>(),
back_inserter( values ) );

return 0;
}

This was created with a little help from
http://www.sgi.com/tech/stl/istream_iterator.html.

Mike

re: q: fast file IO

"Mike Austin" <no@spam.com> wrote in message
news:ziEde.174292$cg1.67616@bgtnsc04-news.ops.worldnet.att.net...[color=blue]
> laniik wrote:[color=green]
>> Hi. I am trying to read in a large number of floats from an ASCII
>> file.
>>
>> I am currently using i/o streams as such:
>>
>> vector <float> values;
>> float val;
>> ifstream in(fn);
>> in>>height;
>> in>>width;
>>
>> for(int i=0;i<height;i++) {
>> for(int j=0;j<width;j++) {
>>
>> in>>val;
>> values.push_back(val);
>>
>> }
>> }
>>
>> where the data is:
>>
>> Width Height
>> value value value value value value...
>>
>>
>> However, this is turning out to be really slow when I read about a
>> million points. I was wondering if anyone knows how i could read this
>> data in much faster? I know that it is possible because several
>> applications do it already Maybe using C FILEs is faster than streams?
>> Im not sure.
>>
>> Thanks!
>>
>> Oliver[/color]
>
> You could try this:
>
> #include <vector>
> #include <fstream>
> #include <iterator>
>
> using namespace std;
>
> int main() {
> ifstream file( filename );
> int width, height;
>
> file >> width >> height;
> vector<float> values( width * height );
>
> copy( istream_iterator<float>( file ), istream_iterator<float>(),
> back_inserter( values ) );
>
> return 0;[/color]

This is unlikely to improve performance, since 'istream_iterator'
uses operator>>(), which 'lanik' was already using. 'lanik's idea
of using a FILE* instead of an ifstream might or might not be
faster, the only way to know is to measure. Another possibility
is to use some nonstandard platform-specific method which might
exploit a more intimate connection with the OS/file system.

-Mike

re: q: fast file IO

It is likely that the slowness is mainly caused by the repeatedly
reallocations of your vector object, rather than file operations. You
may try reserve() to pre-allocate the amount of memory expected.

re: q: fast file IO

Zenith wrote:
[color=blue]
> It is likely that the slowness is mainly caused by the repeatedly
> reallocations of your vector object, rather than file operations. You
> may try reserve() to pre-allocate the amount of memory expected.[/color]

You can also try a std::list. It wont double in size as a vector tends to
do.

Alvin

--
Why must I click 'Start' in order to turn off my computer?

re: q: fast file IO

Mike Wahler wrote:[color=blue]
> This is unlikely to improve performance, since 'istream_iterator'
> uses operator>>(), which 'lanik' was already using. 'lanik's idea
> of using a FILE* instead of an ifstream might or might not be
> faster, the only way to know is to measure. Another possibility
> is to use some nonstandard platform-specific method which might
> exploit a more intimate connection with the OS/file system.
>
> -Mike[/color]

Let's try it another way then. If each float string is a specific
length, we can take advantage of that:

#include <iostream>
#include <vector>
#include <iterator>
#include <fstream>

using namespace std;

const int readLength = 9;

int main( int argc, char* argv[] ) {
ifstream file( "float_data" );
int width, height, size;
char buffer[256];

file >> width >> height;
size = width * height
vector<float> values( size );

for( int i = 0; i < size && file.good(); ++i ) {
file.get( buffer, readLength );
values[i] = atof( buffer );
}

vector<float>::iterator i;
for( i = values.begin(); i != values.end(); ++i ) {
cout << *i << " " << flush;
}

return 0;
}

Mike

re: q: fast file IO

i dont know this for sure, but I have heard that the atof function is
fairly slow. and that i could speed up reading time by writing my own
atof function. so maybe using .get() with my own atof will be faster
than <<.

also, does anyone know how << turns what it reads in into a float?
wonder how fast that part is.

ill try the memory allocation idea, ill try that, that makes sense.

the list idea might speed things up, but I need to have random access
abilities with this array.

thanks everyone!

re: q: fast file IO

laniik wrote:[color=blue]
> i dont know this for sure, but I have heard that the atof function is
> fairly slow. and that i could speed up reading time by writing my own
> atof function. so maybe using .get() with my own atof will be faster
> than <<.
>
> also, does anyone know how << turns what it reads in into a float?
> wonder how fast that part is.
>
> ill try the memory allocation idea, ill try that, that makes sense.
>
> the list idea might speed things up, but I need to have random access
> abilities with this array.
>
> thanks everyone![/color]

I've heard things like this often. The next question would be, why
provide library functions if they are useless and slow? One factor is
of course that the library routines are very general. Here's an
implementation I found with google:

http://www.jbox.dk/sanos/source/lib/strtod.c.html

The scaling loop could be eliminated in trade for accuracy:

int scale = 1;

[...]

if (*p == '.') {
p++;
while (isdigit(*p)) {
number = number * 10. + (*p - '0');
p++;
scale *= 10;
}
}

[...]

return number / scale;


Regards,
Mike

re: q: fast file IO

If you have the option of changing the input file format, try storing
binary data. Use something like this:

union{float num;char data[sizeof(num)];};

That way the big killer (converting floats to and from strings) gets
killed!

Samee