Problems using zlib - buffer problem or something else?

[patrickdepinguin]

Hi,

I use zlib to write data structures to a compressed file, using the
gzwrite function. Afterwards I read the data back with gzread. I notice
that this works well when the data written is not that much, but when
there is more data to write, after a while I get data errors when
reading back the data.
Error in main: couldn't read stat
zlib error -3: test512-20070531-18h10m02.stat.gz: data error


I wonder whether I am doing something wrong. The data I write are simply
C data structures, struct acmp_stat, which have a fixed size.

I am using my internal buffer, after noticing that it helped the
problem. However, now, even though I buffer I get problems. Writing is
done with:

if (data->buflen + sizeof(*s) > data->bufmaxlen)
        if (0 != acmp_flush_buffer(data))
            printf("Error: could not flush buffer\n");
 
    acmp_buffer_stat(data);

Data being:

struct acmp_data {
    void *fd;
    void *buf;
    size_t buflen;
    size_t bufmaxlen;
 
    char *statpath;
 
    struct acmp_stat **stat;
};

The flush buffer function is:

static int
acmp_flush_buffer(struct acmp_data *data)
{
    size_t len;
 
    assert(NULL != data);
    assert(NULL != data->fd);
 
    len = (size_t)gzwrite((gzFile)data->fd, data->buf, data->buflen);
 
    if (len != data->buflen)
        return 1;
 
    data->buflen = 0;
    return 0;
}

and acmp_buffer_stat simply:

static void
acmp_buffer_stat(struct acmp_data *data)
{
    assert(NULL != data);
 
    memcpy(data->buf + data->buflen, data->stat[data->curcpu], sizeof(**data->stat));
    data->buflen += sizeof(**data->stat);
}

Of course I have opened and closed the file before and after using this
code. The buffer is currently assigned a size of 1 MiB.

Reading back the data is done with:

static int
acmp_read_stat(/*@out@*/ struct acmp_stat *stat, void *fd)
{
    int err;
    const char *errmsg;
    size_t len;
 
    assert(NULL != fd);
    assert(stat != NULL);
 
    len = (size_t)gzread((gzFile)fd, (void *)stat, sizeof(*stat));
    errmsg = gzerror((gzFile)fd, &err);
    if (0 > err)
        printf("zlib error %d: %s \n", err, errmsg);
 
    if (sizeof(*stat) != len)
        return 1;
 
    return 0;
}

The code looks ok to me, and the weird thing is that I only get errors
when the amount of data written is large. I didn't find a clear point
yet, from where it seems to fail.
Also notice that there are no errors when writing, but only when reading
back. The files are not touched between reading and writing, and are
cleanly closed.

Am I doing something wrong here?

Thanks a lot, Thomas

[xinunus]

Not sure if this is your problem but zlib does state that your source buffer must be at least 0.1 times larger than the source buffer plus 12 bytes to cope with the overhead of zlib data streams.

[xinunus]

Check out this example:
http://www.gamedev.net/reference/articles/article2279.asp