Need help reading file data

[Tom]

I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
.. . .
.. . .
.. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");

for(i=0;i<=499;i++)
{
fscanf(fp,"%f",&(a[i]));

for(j=1;j<=10002;j++)
{
fscanf(fp,"%f",&b);
}

Dosomethingwith(float a, 500);
}
---------------------------------------------------------------------------------------------

[Thad Smith]

Tom wrote:

I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
. . .
. . .
. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");

for(i=0;i<=499;i++)
{
fscanf(fp,"%f",&(a[i]));

for(j=1;j<=10002;j++)
{
fscanf(fp,"%f",&b);
}

Dosomethingwith(float a, 500);
}
---------------------------------------------------------------------------------------------

If the data is in strict character columns you may be able to seek to
the exact positions of the file where each datum is. I believe it is
not guaranteed for text files, but usually works.

Another approach is to transpose the original data file before processing.

--
Thad

[Charlie Gordon]

"Tom" <he*****@gmail.coma écrit dans le message de news:
11**********************@k79g2000hse.googlegroups. com...

>I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
. . .
. . .
. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

It should not: the amount of data is roughly 5 million values, which can be
loaded in memory as float or double for a total of 20 or 40 megabytes. If
you are running XP, you must have a lot more RAM than that. It could also
be a limitation of the C compiler you are using. Try cygwin and gcc.

All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");

for(i=0;i<=499;i++)

Better use the actual number of elements:

for (i = 0; i < 500; i++)

{
fscanf(fp,"%f",&(a[i]));

for(j=1;j<=10002;j++)
{
fscanf(fp,"%f",&b);
}

try removing this second loop and just scan for the '\n':

{
int c;
while ((c = getc(fp)) != EOF && c != '\n')
continue;
}

This should make the program faster.

>
Dosomethingwith(float a, 500);
}
---------------------------------------------------------------------------------------------

--
Chqrlie.

[Barry Schwarz]

On Sun, 30 Sep 2007 17:49:44 -0000, Tom <he*****@gmail.comwrote:

>I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
. . .
. . .
. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

What error? Compile time or run time?

>
All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");

You should check that fopen succeeded.

>
for(i=0;i<=499;i++)
{
fscanf(fp,"%f",&(a[i]));

You should check that fscanf succeeded. You have now read the first
column of a record.

>
for(j=1;j<=10002;j++)

This loop will try to read the next 10002 columns of the record.
Unfortunately, there are only 10001 columns left to be read. At the
end of this loop you will have read the first column of the next
record.

You might want to try using fgets and strtod to insure you never lose
synchronization.

> {
fscanf(fp,"%f",&b);
}

Dosomethingwith(float a, 500);

It seems like you should be using a[i] and b here.

> }
---------------------------------------------------------------------------------------------

Remove del for email

[alex]

On Oct 1, 1:49 am, Tom <[e-mail]:

I have a datafile which has a form of

10.10 10.20 10.30 ...
20.10 20.20 20.30 ...
30.10 30.20 30.30 ...
. . .
. . .
. . .

it has 500 rows and 10002 colums.

I tried to read the whole data in a single variable.
My pc shows error. (Windows XP, notebook)

All I need is to read each column data and do some integration.
and so on for every colum.

my naive code is following but, it is too slow.
(it is also limited to read first column, but have to expand to read
every column)

---------------------------------------------------------------------------------------------
FILE *fp;
float b;
int i,j;
float a[500];

fp = fopen("2Dwedge_V1_Ga2.36_st6.28_K1.06_B4.00_e0.20p osi.dat","r");

for(i=0;i<=499;i++)
{
fscanf(fp,"%f",&(a[i]));

for(j=1;j<=10002;j++)
{
fscanf(fp,"%f",&b);
}

Dosomethingwith(float a, 500);
}
---------------------------------------------------------------------------------------------

Since only numeric data presents, my suggestion is first convert this
text file into binary, and then seek to right
position as needed.

for example,
position(i, j) = i * 10002 * sizeof(float) + j * sizeof(float)
(i, j) is (row, col) index, starting from 0