468,754 Members | 1,387 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,754 developers. It's quick & easy.

Reading large files

JS
Hello all!

I have come on to a problem for which I am not able to find a solution
searching the web.

What I am trying to do is reading a log-file with a size of 1.3 GB.
When reading it using fread() or the Visual C specific(?) read() they
return that the number of bytes read equals the number of bytes
requested, however all of the bytes read turn out to have a value of
zero (the file contains other values). When using the exact same code
on a much smaller file (1 MB) it works fine.

char filename[] = "c:/temp/data2000.dat";
char buffer[40];
FILE *file_d;
struct _stat stat_buf;
int fd;
int file_exists;
long file_size;

//Returns true:
file_exists = (_stat(filename,&stat_buf)==0) && (stat_buf.st_mode &
S_IFREG);
//Returns 1328000000 (actual filesize):
file_size = (file_exists) ? stat_buf.st_size : -1;
//Booth returns values <> NULL and -1:
//file_d = fopen(filename, "r");
fd = open(filename, _O_RDONLY);

//Both returns 40:
//bytes_read = fread(buffer, 1, sizeof(buffer), file_d);
bytes_read = read(fd, buffer, 40);

//fclose(file_d);
close(fd);

Greatful for any ideas!
Best regards
Nov 13 '05 #1
2 14874
mi*********@hotmail.com (JS) wrote:
Hello all!

I have come on to a problem for which I am not able to find a solution
searching the web.

What I am trying to do is reading a log-file with a size of 1.3 GB.
When reading it using fread() or the Visual C specific(?) read() they
return that the number of bytes read equals the number of bytes
requested, however all of the bytes read turn out to have a value of
zero (the file contains other values). When using the exact same code
on a much smaller file (1 MB) it works fine.


<CODE SNIPPED>

Did you make sure your file contains pure text data? If it does not,
you should open it in binary mode.

Note that _stat(), open(), read(), and close() are not standard C
functions (Ah, I see from your remark above you already know ...).

Also note that both of us swapped the second and third argument of
fread, though this is useful to get the number of characters read in the
final call to fread, which will result most certainly in a partial
filled buffer. (It doesn't really matter anyway in this case, because
there's not a big difference between fread performing 40*1 or 1*40 calls
to fgetc respectively.)

The program below works fine for me (tested with a 2.8 GB file):

#include <stdio.h>
#include <stdlib.h>

#define BUFELEM 40
#define FILENAME "test.dat"

int main( void )
{
size_t num;
unsigned long count;
unsigned char buf[ BUFELEM ];
FILE *fp;

if ( ( fp = fopen( FILENAME, "rb" ) ) != NULL )
{
count = 0;
do
{
num = fread( buf, 1, BUFELEM, fp );
++count;
/* do sth. with buffer contents here [1] */
}
while ( num == BUFELEM );

/* Error checking snipped for brevity */

fclose( fp );
printf( "calls to fread: %ld\n", count );
}
else
{
fprintf( stderr, "Error opening file %s\n", FILENAME );
return EXIT_FAILURE;
}

return EXIT_SUCCESS;
}

[1] You may write the buffer contents to another file and compare
it to the original to make sure valid data had ben read.
Regards

Irrwahn
--
The generation of random numbers is too important
to be left to chance.
Nov 13 '05 #2
On 19 Sep 2003 00:13:37 -0700, mi*********@hotmail.com (JS) wrote:
Hello all!

I have come on to a problem for which I am not able to find a solution
searching the web.

What I am trying to do is reading a log-file with a size of 1.3 GB.
When reading it using fread() or the Visual C specific(?) read() they
return that the number of bytes read equals the number of bytes
requested, however all of the bytes read turn out to have a value of
zero (the file contains other values). When using the exact same code
on a much smaller file (1 MB) it works fine.


<code snipped>

Have you verified that the beginning of the file is not a bunch of
null characters, by using a binary dump facility or something?

- Sev
--
I am just a thought of mine, an egotisticality. (P. Crews)
Nov 13 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

19 posts views Thread by Brad Tilley | last post: by
3 posts views Thread by Michael | last post: by
3 posts views Thread by Paul Spielvogel | last post: by
3 posts views Thread by Eric Twietmeyer | last post: by
2 posts views Thread by Kevin Ar18 | last post: by
17 posts views Thread by byte8bits | last post: by
1 post views Thread by CARIGAR | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.