473,231 Members | 1,621 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,231 software developers and data experts.

Reading large files

JS
Hello all!

I have come on to a problem for which I am not able to find a solution
searching the web.

What I am trying to do is reading a log-file with a size of 1.3 GB.
When reading it using fread() or the Visual C specific(?) read() they
return that the number of bytes read equals the number of bytes
requested, however all of the bytes read turn out to have a value of
zero (the file contains other values). When using the exact same code
on a much smaller file (1 MB) it works fine.

char filename[] = "c:/temp/data2000.dat";
char buffer[40];
FILE *file_d;
struct _stat stat_buf;
int fd;
int file_exists;
long file_size;

//Returns true:
file_exists = (_stat(filename,&stat_buf)==0) && (stat_buf.st_mode &
S_IFREG);
//Returns 1328000000 (actual filesize):
file_size = (file_exists) ? stat_buf.st_size : -1;
//Booth returns values <> NULL and -1:
//file_d = fopen(filename, "r");
fd = open(filename, _O_RDONLY);

//Both returns 40:
//bytes_read = fread(buffer, 1, sizeof(buffer), file_d);
bytes_read = read(fd, buffer, 40);

//fclose(file_d);
close(fd);

Greatful for any ideas!
Best regards
Nov 13 '05 #1
2 15993
mi*********@hotmail.com (JS) wrote:
Hello all!

I have come on to a problem for which I am not able to find a solution
searching the web.

What I am trying to do is reading a log-file with a size of 1.3 GB.
When reading it using fread() or the Visual C specific(?) read() they
return that the number of bytes read equals the number of bytes
requested, however all of the bytes read turn out to have a value of
zero (the file contains other values). When using the exact same code
on a much smaller file (1 MB) it works fine.


<CODE SNIPPED>

Did you make sure your file contains pure text data? If it does not,
you should open it in binary mode.

Note that _stat(), open(), read(), and close() are not standard C
functions (Ah, I see from your remark above you already know ...).

Also note that both of us swapped the second and third argument of
fread, though this is useful to get the number of characters read in the
final call to fread, which will result most certainly in a partial
filled buffer. (It doesn't really matter anyway in this case, because
there's not a big difference between fread performing 40*1 or 1*40 calls
to fgetc respectively.)

The program below works fine for me (tested with a 2.8 GB file):

#include <stdio.h>
#include <stdlib.h>

#define BUFELEM 40
#define FILENAME "test.dat"

int main( void )
{
size_t num;
unsigned long count;
unsigned char buf[ BUFELEM ];
FILE *fp;

if ( ( fp = fopen( FILENAME, "rb" ) ) != NULL )
{
count = 0;
do
{
num = fread( buf, 1, BUFELEM, fp );
++count;
/* do sth. with buffer contents here [1] */
}
while ( num == BUFELEM );

/* Error checking snipped for brevity */

fclose( fp );
printf( "calls to fread: %ld\n", count );
}
else
{
fprintf( stderr, "Error opening file %s\n", FILENAME );
return EXIT_FAILURE;
}

return EXIT_SUCCESS;
}

[1] You may write the buffer contents to another file and compare
it to the original to make sure valid data had ben read.
Regards

Irrwahn
--
The generation of random numbers is too important
to be left to chance.
Nov 13 '05 #2
On 19 Sep 2003 00:13:37 -0700, mi*********@hotmail.com (JS) wrote:
Hello all!

I have come on to a problem for which I am not able to find a solution
searching the web.

What I am trying to do is reading a log-file with a size of 1.3 GB.
When reading it using fread() or the Visual C specific(?) read() they
return that the number of bytes read equals the number of bytes
requested, however all of the bytes read turn out to have a value of
zero (the file contains other values). When using the exact same code
on a much smaller file (1 MB) it works fine.


<code snipped>

Have you verified that the beginning of the file is not a bunch of
null characters, by using a binary dump facility or something?

- Sev
--
I am just a thought of mine, an egotisticality. (P. Crews)
Nov 13 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

19
by: Brad Tilley | last post by:
I have some large files (between 2 & 4 GB) that I want to do a few things with. Here's how I've been using the md5 module in Python: original = file(path + f, 'rb') data = original.read(4096)...
5
by: Paul | last post by:
I'm using Microsoft's ASPFileUpload routines and all works fine as long as my files are smaller than about 200K. For anything larger than about 210K, I get the following error: Error Type:...
3
by: Steven Burn | last post by:
The application; Service on my webserver that allows a user to upload their HOSTS file for functions to verify the contents are still valid. Uses; 1. XMLHTTP (MSXML2) 2. FileSystemObject...
3
by: Michael | last post by:
X-Replace-Address Hello, I am trying to write a program at work for reading/writing files larger than 4 GB. I know that Windows supports files that big but I have not been able to get my...
3
by: Paul Spielvogel | last post by:
I need to compute the MD5 hash on VERY large files 500mb to 4gb+ I have found two ways but neither one of them does what i need. Private Function ComputeDataMD5(ByVal path As String) As String...
3
by: Eric Twietmeyer | last post by:
Hello, I have code written using the basic fstream object on a Win32 system. This is of course simply a typedef for basic_fstream<char, char_traits<char. This object can not be used to write...
1
by: akalmand | last post by:
Hi there, I am writing a code to read some data from the text files. The number of text files is not fixed and could be more that 15. the length of each file is large... close to 100,000 on an...
2
by: Kevin Ar18 | last post by:
I posted this on the forum, but nobody seems to know the solution: http://python-forum.org/py/viewtopic.php?t=5230 I have a zip file that is several GB in size, and one of the files inside of it...
17
by: byte8bits | last post by:
How does C++ safely open and read very large files? For example, say I have 1GB of physical memory and I open a 4GB file and attempt to read it like so: #include <iostream> #include <fstream>...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: abbasky | last post by:
### Vandf component communication method one: data sharing ​ Vandf components can achieve data exchange through data sharing, state sharing, events, and other methods. Vandf's data exchange method...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.