By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
446,305 Members | 1,614 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 446,305 IT Pros & Developers. It's quick & easy.

Problems using zlib - buffer problem or something else?

P: 6

I use zlib to write data structures to a compressed file, using the
gzwrite function. Afterwards I read the data back with gzread. I notice
that this works well when the data written is not that much, but when
there is more data to write, after a while I get data errors when
reading back the data.
Error in main: couldn't read stat
zlib error -3: test512-20070531-18h10m02.stat.gz: data error

I wonder whether I am doing something wrong. The data I write are simply
C data structures, struct acmp_stat, which have a fixed size.

I am using my internal buffer, after noticing that it helped the
problem. However, now, even though I buffer I get problems. Writing is
done with:

Expand|Select|Wrap|Line Numbers
  1. if (data->buflen + sizeof(*s) > data->bufmaxlen)
  2.         if (0 != acmp_flush_buffer(data))
  3.             printf("Error: could not flush buffer\n");
  5.     acmp_buffer_stat(data);
Data being:
Expand|Select|Wrap|Line Numbers
  1. struct acmp_data {
  2.     void *fd;
  3.     void *buf;
  4.     size_t buflen;
  5.     size_t bufmaxlen;
  7.     char *statpath;
  9.     struct acmp_stat **stat;
  10. };
The flush buffer function is:

Expand|Select|Wrap|Line Numbers
  1. static int
  2. acmp_flush_buffer(struct acmp_data *data)
  3. {
  4.     size_t len;
  6.     assert(NULL != data);
  7.     assert(NULL != data->fd);
  9.     len = (size_t)gzwrite((gzFile)data->fd, data->buf, data->buflen);
  11.     if (len != data->buflen)
  12.         return 1;
  14.     data->buflen = 0;
  15.     return 0;
  16. }

and acmp_buffer_stat simply:

Expand|Select|Wrap|Line Numbers
  1. static void
  2. acmp_buffer_stat(struct acmp_data *data)
  3. {
  4.     assert(NULL != data);
  6.     memcpy(data->buf + data->buflen, data->stat[data->curcpu], sizeof(**data->stat));
  7.     data->buflen += sizeof(**data->stat);
  8. }

Of course I have opened and closed the file before and after using this
code. The buffer is currently assigned a size of 1 MiB.

Reading back the data is done with:

Expand|Select|Wrap|Line Numbers
  1. static int
  2. acmp_read_stat(/*@out@*/ struct acmp_stat *stat, void *fd)
  3. {
  4.     int err;
  5.     const char *errmsg;
  6.     size_t len;
  8.     assert(NULL != fd);
  9.     assert(stat != NULL);
  11.     len = (size_t)gzread((gzFile)fd, (void *)stat, sizeof(*stat));
  12.     errmsg = gzerror((gzFile)fd, &err);
  13.     if (0 > err)
  14.         printf("zlib error %d: %s \n", err, errmsg);
  16.     if (sizeof(*stat) != len)
  17.         return 1;
  19.     return 0;
  20. }

The code looks ok to me, and the weird thing is that I only get errors
when the amount of data written is large. I didn't find a clear point
yet, from where it seems to fail.
Also notice that there are no errors when writing, but only when reading
back. The files are not touched between reading and writing, and are
cleanly closed.

Am I doing something wrong here?

Thanks a lot, Thomas
Jun 3 '07 #1
Share this Question
Share on Google+
2 Replies

P: 3
Not sure if this is your problem but zlib does state that your source buffer must be at least 0.1 times larger than the source buffer plus 12 bytes to cope with the overhead of zlib data streams.
Nov 14 '07 #2

P: 3
Check out this example:
Nov 14 '07 #3

Post your reply

Sign in to post your reply or Sign up for a free account.