By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
445,813 Members | 1,257 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 445,813 IT Pros & Developers. It's quick & easy.

fwrite() misses writing a byte

P: n/a
I have an application which writes huge number of bytes into the
binary files which is just some marshalled data.

int len = Data.size(); //arrary size
for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);

now after running this for long time and pushing millions of bytes, It
once misses writing the last byte of fData. Then the further push of
bytes is again correct. As i am not using the return value for the
fwrite I think there is some error which could be due to buffer
overrun in the stream. This is hard to reproduce and happening on the
Solaris.
Any help will be appreciated.

Oct 3 '07 #1
Share this Question
Share on Google+
12 Replies


P: n/a
he*********@gmail.com wrote:
I have an application which writes huge number of bytes into the
binary files which is just some marshalled data.

int len = Data.size(); //arrary size
You seem to using C++. If so then <news:comp.lang.c++might be a better
venue for your question.
for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);
Can't you write more than one byte at a time? This is probably an
inefficient method for writing large number of bytes. Also consider using
C++ fstream object for this, since you're using C++ anyway.
now after running this for long time and pushing millions of bytes, It
once misses writing the last byte of fData. Then the further push of
bytes is again correct. As i am not using the return value for the
fwrite I think there is some error which could be due to buffer
overrun in the stream. This is hard to reproduce and happening on the
Solaris.
Any help will be appreciated.
There is no excuse to ignore the status values that these I/O functions
report except perhaps in throwaway programs. fwrite returns the number of
data items successfully written, so a shorter count than expected means
that there was an error. The most likely cause is that the disk quota was
exceeded.

Anyway you should repost this to comp.lang.c++, since you're using C++.

Oct 3 '07 #2

P: n/a
he*********@gmail.com wrote:
I have an application which writes huge number of bytes into the
binary files which is just some marshalled data.

int len = Data.size(); //arrary size
for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);

now after running this for long time and pushing millions of bytes, It
once misses writing the last byte of fData. Then the further push of
bytes is again correct. As i am not using the return value for the
fwrite I think there is some error which could be due to buffer
overrun in the stream. This is hard to reproduce and happening on the
Solaris.
Any help will be appreciated.
It would be a good idea to TEST THE RETURN VALUE OF fwrite!!!

Why you don't do that instead of guessing and posting into
newsgroups?
--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32
Oct 3 '07 #3

P: n/a
<he*********@gmail.comschrieb im Newsbeitrag
news:11*********************@57g2000hsv.googlegrou ps.com...
>I have an application which writes huge number of bytes into the
binary files which is just some marshalled data.

int len = Data.size(); //arrary size
for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);

now after running this for long time and pushing millions of bytes, It
once misses writing the last byte of fData. Then the further push of
bytes is again correct. As i am not using the return value for the
fwrite I think there is some error which could be due to buffer
overrun in the stream. This is hard to reproduce and happening on the
Solaris.
Any help will be appreciated.
You'd need to check the return value of fwrite(). It it's not equal to it's
3rd parameter, something went wrong.
You may increase performance of your writes by not doing it byte by byte.
size_t towrite = Data.size();
size_t written = fwrite(Data, 1, towrite, f);
if (written != towrite)
fputs("Whoops!\n", stderr);
Your code looks like C++ to me, so you may want to ask in comp.lang.c++

Bye, Jojo
Oct 3 '07 #4

P: n/a
santosh said:
he*********@gmail.com wrote:
>I have an application which writes huge number of bytes into the
binary files which is just some marshalled data.

int len = Data.size(); //arrary size

You seem to using C++.
The line you quoted sure looks like C to me (C99, admittedly).
> for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);

Can't you write more than one byte at a time? This is probably an
inefficient method for writing large number of bytes.
How can you tell? Have you measured - and on *his* system? If the stream is
buffered, it won't make so much as a halfpenny-worth of difference whether
he writes one object at a time or all len objects at once.

<snip>

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 3 '07 #5

P: n/a
Richard Heathfield wrote:
santosh said:
> he*********@gmail.com wrote:
>>I have an application which writes huge number of bytes into the
binary files which is just some marshalled data.

int len = Data.size(); //arrary size

You seem to using C++.

The line you quoted sure looks like C to me (C99, admittedly).
>> for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);

Can't you write more than one byte at a time? This is probably an
inefficient method for writing large number of bytes.

How can you tell? Have you measured - and on *his* system? If the stream
is buffered, it won't make so much as a halfpenny-worth of difference
whether he writes one object at a time or all len objects at once.

<snip>
You are right. However it seemed like an inefficient method when a single
fwrite call could do the same job. As you note _if_ the stream is buffered,
there wouldn't be any difference worth noting between the two methods, but
there is a small penalty to be paid for making N calls to fwrite instead of
one call. Below are the measurements on my machine. As you note, the
difference in negligible.

$ ./fwrt_test
Usage: ./fwrt_test N
If N is 0, single fwrite call
If N is 1, one fwrite call for each data element
$ ./fwrt_test 0
wrote 40000000 bytes (10000000 items, 4 bytes/item)
time taken: 0.120000
$ ./fwrt_test 1
wrote 40000000 bytes (10000000 items, 4 bytes/item), one item at a time.
time taken: 0.630000
$ ./fwrt_test 0
wrote 40000000 bytes (10000000 items, 4 bytes/item)
time taken: 0.120000
$ ./fwrt_test 1
wrote 40000000 bytes (10000000 items, 4 bytes/item), one item at a time.
time taken: 0.600000
$ ./fwrt_test 0
wrote 40000000 bytes (10000000 items, 4 bytes/item)
time taken: 0.120000
$ ./fwrt_test 1
wrote 40000000 bytes (10000000 items, 4 bytes/item), one item at a time.
time taken: 0.630000

code follows:

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

#define N_ELEMENTS 10000000UL /* ten million elements */

void usage(char *prg_name)
{
fprintf(stdout, "Usage: %s N\nIf N is 0, single fwrite call\n"
"If N is 1, one fwrite call for each data element\n",
prg_name
);
return;
}

int main(int argc, char **argv)
{
char choice;
int *buf;
size_t bsize, ctr;
clock_t begin, end;
FILE *fp;

if (argc < 2) {
usage(argv[0]);
exit(EXIT_FAILURE);
}
else {
choice = argv[1][0];
if (choice != '0' && choice != '1') {
usage(argv[0]);
exit(EXIT_FAILURE);
}
}

/* allocate buffer, fill with pseudo-random integers */
bsize = N_ELEMENTS * sizeof *buf;
if ((buf = malloc(bsize)) == NULL) {
fprintf(stderr, "malloc failed - %lu bytes.\n", (unsigned
long)bsize);
exit(EXIT_FAILURE);
}
else
for (ctr = 0; ctr < N_ELEMENTS; ctr++) buf[ctr] = rand();

/* open a disk file */
if ((fp = fopen("outfile", "w")) == NULL) {
fprintf(stderr, "fopen failed.\n");
free(buf);
exit(EXIT_FAILURE);
}

/* write to file either at once or one element at a time; time it. */
if (choice == '0') {
begin = clock();
if (fwrite(buf, sizeof *buf, N_ELEMENTS, fp) != N_ELEMENTS) {
fprintf(stderr, "fwrite failed.\n");
fclose(fp);
free(buf);
exit(EXIT_FAILURE);
}
end = clock();
fprintf(stdout, "wrote %lu bytes (%lu items, %lu bytes/item)\n"
"time taken: %f\n", (unsigned long)bsize, N_ELEMENTS,
(unsigned long)sizeof *buf,
(double)(end - begin) / CLOCKS_PER_SEC
);
}
else {
begin = clock();
for (ctr = 0; ctr < N_ELEMENTS; ctr++) {
if (fwrite(buf+ctr, sizeof *buf, 1, fp) != 1) {
fprintf(stderr, "fwrite failed.\n");
fclose(fp);
free(buf);
exit(EXIT_FAILURE);
}
}
end = clock();
fprintf(stdout, "wrote %lu bytes (%lu items, %lu bytes/item), one"
" item at a time.\ntime taken: %f\n", (unsigned long)bsize,
N_ELEMENTS, (unsigned long)sizeof *buf,
(double)(end - begin) / CLOCKS_PER_SEC
);
}
fclose(fp);
free(buf);
return EXIT_SUCCESS;
}
Oct 3 '07 #6

P: n/a
Richard Heathfield <rj*@see.sig.invalidwrites:
santosh said:
> he*********@gmail.com wrote:
>>I have an application which writes huge number of bytes into the
binary files which is just some marshalled data.

int len = Data.size(); //arrary size

You seem to using C++.

The line you quoted sure looks like C to me (C99, admittedly).
>> for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);
[...]

The OP didn't show us the declaration of Data, but apparently it has a
member named "size" that's a function pointer *and* it can be indexed
with the [] operator. Either he's using operator overloading (most
likely in C++, so "size" is probably only implicitly a function
pointer) or he hasn't shown us his actual code.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
Oct 3 '07 #7

P: n/a
Keith Thompson said:

<snip>
The OP didn't show us the declaration of Data, but apparently it has a
member named "size" that's a function pointer *and* it can be indexed
with the [] operator.
Was your college nickname "Hawkeye"? That was an excellent spot, and it
does indeed suggest that the OP is after all using Another Language.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 4 '07 #8

P: n/a
On Wed, 03 Oct 2007 13:39:01 -0700, Keith Thompson wrote:
You should also decide whether you're using C or C++. Your use of
both ``Data.size()'' and ``Data[i]'' suggests that you're using
operator overloading, which C++ supports but C doesn't.
static unsigned char foo[1000];
static int bar(void) { return sizeof foo; }
struct { int (*size)(void); } baz = { &bar };
#define Data (__LINE__ % 3 ? baz : foo)
;-)

--
Army1987 (Replace "NOSPAM" with "email")
A hamburger is better than nothing.
Nothing is better than eternal happiness.
Therefore, a hamburger is better than eternal happiness.

Oct 5 '07 #9

P: n/a
Army1987 <ar******@NOSPAM.itwrites:
On Wed, 03 Oct 2007 13:39:01 -0700, Keith Thompson wrote:
>You should also decide whether you're using C or C++. Your use of
both ``Data.size()'' and ``Data[i]'' suggests that you're using
operator overloading, which C++ supports but C doesn't.
static unsigned char foo[1000];
static int bar(void) { return sizeof foo; }
struct { int (*size)(void); } baz = { &bar };
#define Data (__LINE__ % 3 ? baz : foo)
;-)
Clever, but it's a constraint violation; see C99 6.5.15 for the
requirements on the second and third operands of the conditional
operator.

--
Keith Thompson (The_Other_Keith) ks***@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
Oct 5 '07 #10

P: n/a
On Wed, 03 Oct 2007 12:33:07 +0200, Joachim Schmitz wrote:
<he*********@gmail.comschrieb im Newsbeitrag
news:11*********************@57g2000hsv.googlegrou ps.com...
>>I have an application which writes huge number of bytes into the
binary files which is just some marshalled data.

int len = Data.size(); //arrary size
for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);
[...]
Your code looks like C++ to me, so you may want to ask in comp.lang.c++
Replacing Data.size() with a constant and using a suitable
declaration for Data the code is valid C99. Also, the only draft
of the C++ standard I've seen (and probably IMO also the real one)
defines fwrite and the rest of the library as by textual inclusion
of clause 7 of the C standard. So, unless there is something wrong
with Data::size(), that's not actually a C++ problem.
(Anyway, we don't see enough code to tell what's wrong, so the
only topical answer here is "probably there's a IO error, check
the result of fwrite", and given the OP's description of the
problem it is not very unlikely.)
--
Army1987 (Replace "NOSPAM" with "email")
A hamburger is better than nothing.
Nothing is better than eternal happiness.
Therefore, a hamburger is better than eternal happiness.

Oct 7 '07 #11

P: n/a
"Richard Heathfield" <rj*@see.sig.invalida écrit dans le message de news:
gN******************************@bt.com...
santosh said:
> he*********@gmail.com wrote:
>>I have an application which writes huge number of bytes into the
binary files which is just some marshalled data.

int len = Data.size(); //arrary size

You seem to using C++.

The line you quoted sure looks like C to me (C99, admittedly).
>> for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);

Can't you write more than one byte at a time? This is probably an
inefficient method for writing large number of bytes.

How can you tell? Have you measured - and on *his* system? If the stream
is
buffered, it won't make so much as a halfpenny-worth of difference whether
he writes one object at a time or all len objects at once.
It would definitely simplify the code.

--
Chqrlie.
Oct 11 '07 #12

P: n/a
Charlie Gordon said:
"Richard Heathfield" <rj*@see.sig.invalida écrit dans le message de
news: gN******************************@bt.com...
>santosh said:
>> he*********@gmail.com wrote:
<snip>
>>> for (int i = 0; i < len; ++i)
fwrite(&Data[i], 1, 1, f);

Can't you write more than one byte at a time? This is probably an
inefficient method for writing large number of bytes.

[...] If the stream is buffered, it won't make [much] difference [...]

It would definitely simplify the code.
An excellent point.

--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Oct 11 '07 #13

This discussion thread is closed

Replies have been disabled for this discussion.