Reading large files

[JS]

Hello all!

I have come on to a problem for which I am not able to find a solution
searching the web.

What I am trying to do is reading a log-file with a size of 1.3 GB.
When reading it using fread() or the Visual C specific(?) read() they
return that the number of bytes read equals the number of bytes
requested, however all of the bytes read turn out to have a value of
zero (the file contains other values). When using the exact same code
on a much smaller file (1 MB) it works fine.

char filename[] = "c:/temp/data2000.dat";
char buffer[40];
FILE *file_d;
struct _stat stat_buf;
int fd;
int file_exists;
long file_size;

//Returns true:
file_exists = (_stat(filename ,&stat_buf)== 0) && (stat_buf.st_mo de &
S_IFREG);
//Returns 1328000000 (actual filesize):
file_size = (file_exists) ? stat_buf.st_siz e : -1;
//Booth returns values <> NULL and -1:
//file_d = fopen(filename, "r");
fd = open(filename, _O_RDONLY);

//Both returns 40:
//bytes_read = fread(buffer, 1, sizeof(buffer), file_d);
bytes_read = read(fd, buffer, 40);

//fclose(file_d);
close(fd);

Greatful for any ideas!
Best regards

[Irrwahn Grausewitz]

mi*********@hot mail.com (JS) wrote:

Hello all!

I have come on to a problem for which I am not able to find a solution
searching the web.

What I am trying to do is reading a log-file with a size of 1.3 GB.
When reading it using fread() or the Visual C specific(?) read() they
return that the number of bytes read equals the number of bytes
requested, however all of the bytes read turn out to have a value of
zero (the file contains other values). When using the exact same code
on a much smaller file (1 MB) it works fine.

<CODE SNIPPED>

Did you make sure your file contains pure text data? If it does not,
you should open it in binary mode.

Note that _stat(), open(), read(), and close() are not standard C
functions (Ah, I see from your remark above you already know ...).

Also note that both of us swapped the second and third argument of
fread, though this is useful to get the number of characters read in the
final call to fread, which will result most certainly in a partial
filled buffer. (It doesn't really matter anyway in this case, because
there's not a big difference between fread performing 40*1 or 1*40 calls
to fgetc respectively.)

The program below works fine for me (tested with a 2.8 GB file):

#include <stdio.h>
#include <stdlib.h>

#define BUFELEM 40
#define FILENAME "test.dat"

int main( void )
{
size_t num;
unsigned long count;
unsigned char buf[ BUFELEM ];
FILE *fp;

if ( ( fp = fopen( FILENAME, "rb" ) ) != NULL )
{
count = 0;
do
{
num = fread( buf, 1, BUFELEM, fp );
++count;
/* do sth. with buffer contents here [1] */
}
while ( num == BUFELEM );

/* Error checking snipped for brevity */

fclose( fp );
printf( "calls to fread: %ld\n", count );
}
else
{
fprintf( stderr, "Error opening file %s\n", FILENAME );
return EXIT_FAILURE;
}

return EXIT_SUCCESS;
}

[1] You may write the buffer contents to another file and compare
it to the original to make sure valid data had ben read.
Regards

Irrwahn
--
The generation of random numbers is too important
to be left to chance.

[Severian]

On 19 Sep 2003 00:13:37 -0700, mi*********@hot mail.com (JS) wrote:

Hello all!

I have come on to a problem for which I am not able to find a solution
searching the web.

What I am trying to do is reading a log-file with a size of 1.3 GB.
When reading it using fread() or the Visual C specific(?) read() they
return that the number of bytes read equals the number of bytes
requested, however all of the bytes read turn out to have a value of
zero (the file contains other values). When using the exact same code
on a much smaller file (1 MB) it works fine.

<code snipped>

Have you verified that the beginning of the file is not a bunch of
null characters, by using a binary dump facility or something?

- Sev
--
I am just a thought of mine, an egotisticality. (P. Crews)