Corrupted ZIP files during download via PHP

[Swede]

Hello,

I am posting this rather lengthy description here in hope of getting at
least some insight into a problem that is really hurting my business,
since I have come to the conclusion that my problem is at least partly
PHP-related.

Some time now I have been having transfer problems, i.e. my customers
downloads of large binary (ZIP) files are often corrupt.

A corrupted file now and then can of course never be avoided, but I get
several complaints from customers every day. Of any specific file,
approx. 1 out of 4 downloads (larger than 100kB or so) are affected.
I have spent months trying to find the source of the problems without
success, but I have been able to rule out many possible causes. Here are
some facts:

1) The actual files are OK on the server. Direct (unencrypted) links are
fine, so it's not a general problem.

2) About 50% of the customers download (the exact same files) from
another source. I don't even get 1 complaint every two months from those
customers, while I get 1-3 complaints _every day_ from those who
download from my server.

3) The customers have various SW/HW/OS, so it's very unlikely that it's
a user problem. Plus, the same customer often download from both places,
but only have problems with the files from my server.

4) I have shut down all scripts for a while, so that only a static
website remained, but it didn't help.

5) I have had this problem on three servers: The present RaQ550, the
previous RaQ4 and the RaQ4 I had before that. So it's highly unlikely
that it is server-related.

6) I can easily reproduce the problem myself. As I looked into this I
found a systematic error:

Analyzing corrupt vs. clean ZIP file downloads with a binary/hex editor
clearly shows that the error is systematic. Clean ZIP files have a lot
of '00h' bytes in them. In ALL of the corrupt ZIP files ALL (well, most)
of these bytes have been exchanged for '5C30h', i.e. TWO bytes ( 00h ->
5C30h ) !

Since '5C30h' equals an escaped zero, i.e. "\0", my guess is that there
is a translation error somewhere.

So, what does PHP have to do with this? Well, my download links to
purchaseware are encrypted by the PHP "ewportal":
http://www.eliteweaver.co.uk/antifra....php?page=Home

From the few facts I have been able to uncover it seems there is
sometimes a problem with corrupted ZIP files depending on the
setting/compilation of ZLIB. If anyone on this forum have experienced
similar things I would very much appreciate if they could share.
My servers PHP configuration is at:
http://www.snith.com/phptest.php

Best regards, Lars

[Alvaro G. Vicario]

*** Swede escribió/wrote (Tue, 15 Feb 2005 21:16:18 +0100):

Some time now I have been having transfer problems, i.e. my customers
downloads of large binary (ZIP) files are often corrupt.
I suggest that you check the server's MIME type for *.zip files.

From the few facts I have been able to uncover it seems there is
sometimes a problem with corrupted ZIP files depending on the
setting/compilation of ZLIB.

Do you compress the files on the fly?
--
-+ Álvaro G. Vicario - Burgos, Spain
+- http://www.demogracia.com (la web de humor barnizada para la intemperie)
++ Manda tus dudas al grupo, no a mi buzón
-+ Send your questions to the group, not to my mailbox
--

[Swede]

Hello Alvaro,

Alvaro G. Vicario wrote:

*** Swede escribió/wrote (Tue, 15 Feb 2005 21:16:18 +0100):

Some time now I have been having transfer problems, i.e. my customers
downloads of large binary (ZIP) files are often corrupt.

I suggest that you check the server's MIME type for *.zip files.

I have already tried that, it was set to some non-descript application.
I changed it to "octet-stream" together with "*.exe" and the other
binaries, but it did not help. Is there any more place than "mime.types"
where I could change this?

I have seen some brief comments about "commenting out the header which
gets the mimetype of the file in class.download.inc", but I have no idea
how/where to do that, and if it is related to my problem.

From the few facts I have been able to uncover it seems there is
sometimes a problem with corrupted ZIP files depending on the
setting/compilation of ZLIB.

Do you compress the files on the fly?

Not intentionally...they are already ZIPs on the server disk.

Thanks for your input so far, I think you are definitely on the right
path! Any more good ideas?
Best regards, Lars

[Alvaro G. Vicario]

*** Swede escribió/wrote (Tue, 15 Feb 2005 21:52:39 +0100):

I have already tried that, it was set to some non-descript application.
I changed it to "octet-stream" together with "*.exe" and the other
binaries, but it did not help. Is there any more place than "mime.types"
where I could change this?

My suggestion only came because maybe the server was using the default type
(usually text/html) and was messing with character encoding or God knows
what.

Anyway, since you seem to be using a PHP script to serve files then it's
the script the one who should be setting the MIME type; otherwise you'll
get the default type. Something similar to this should do:

header('Content-Type: application/zip');

Internet Explorer is known for having problems with PDF files (esp. if you
use sessions) but I'm not sure about ZIPs.

--
-+ Álvaro G. Vicario - Burgos, Spain
+- http://www.demogracia.com (la web de humor barnizada para la intemperie)
++ Manda tus dudas al grupo, no a mi buzón
-+ Send your questions to the group, not to my mailbox
--

[Swede]

Hello Again,

Alvaro G. Vicario wrote:

*** Swede escribió/wrote (Tue, 15 Feb 2005 21:52:39 +0100):

I have already tried that, it was set to some non-descript application.
I changed it to "octet-stream" together with "*.exe" and the other
binaries, but it did not help. Is there any more place than "mime.types"
where I could change this?
My suggestion only came because maybe the server was using the default type
(usually text/html) and was messing with character encoding or God knows
what.

Anyway, since you seem to be using a PHP script to serve files then it's
the script the one who should be setting the MIME type; otherwise you'll
get the default type. Something similar to this should do:

header('Content-Type: application/zip');

Sounds like a good idea, should this go into "php.ini", or in the script
itself ?

Internet Explorer is known for having problems with PDF files (esp. if you
use sessions) but I'm not sure about ZIPs.

Yes, IE causes a lot of problems, but I see this also on my Mozilla/Firefox.
Best regards, Lars

[chernyshevsky]

The conversion of binary zeros to "\0"s is consistent with
magic_quotes_runtime = On.

[Alvaro G. Vicario]

*** Swede escribió/wrote (Tue, 15 Feb 2005 22:15:08 +0100):

header('Content-Type: application/zip');

Sounds like a good idea, should this go into "php.ini", or in the script
itself ?

Maybe it's a good idea not to mess with this if you are unsure of what it
does. This is just PHP code, not a configuration directive. How to use it
depends on how your script is coded.

Internet Explorer is known for having problems with PDF files (esp. if you
use sessions) but I'm not sure about ZIPs.

Yes, IE causes a lot of problems, but I see this also on my Mozilla/Firefox.

If you use Firefox, please install the "Live HTTP Headers" extension and
find out yourself what MIME type is used.

--
-+ Álvaro G. Vicario - Burgos, Spain
+- http://www.demogracia.com (la web de humor barnizada para la intemperie)
++ Manda tus dudas al grupo, no a mi buzón
-+ Send your questions to the group, not to my mailbox
--

[Swede]

Hello,

Thanks for the tip. Unfortunately all of magic_quotes_gpc,
magic_quotes_runtime, magic_quotes_sybase are "off" in the php.ini, so
this should not be the cause as far as I can understand.

Best regards, Lars

ch***********@hotmail.com wrote:

The conversion of binary zeros to "\0"s is consistent with
magic_quotes_runtime = On.

[Swede]

Hello Alvaro,

Thanks for your input, I really appreciate it!
Unfortunately the script is not my own to start with, I do many
languages but have not got around to PHP yet. I will try it out
(reversible...). But it will have to be tomorrow, it's bedtime for me
here now (Norway)!

Best regards, Lars
Alvaro G. Vicario wrote:

*** Swede escribió/wrote (Tue, 15 Feb 2005 22:15:08 +0100):

header('Content-Type: application/zip');

Sounds like a good idea, should this go into "php.ini", or in the script
itself ?

Maybe it's a good idea not to mess with this if you are unsure of what it
does. This is just PHP code, not a configuration directive. How to use it
depends on how your script is coded.

Internet Explorer is known for having problems with PDF files (esp. if you
use sessions) but I'm not sure about ZIPs.

Yes, IE causes a lot of problems, but I see this also on my Mozilla/Firefox.

If you use Firefox, please install the "Live HTTP Headers" extension and
find out yourself what MIME type is used.

[chernyshevsky]

Don't know if it would help in this case, but Igor Tandetnik's
PassThruAPP is very useful for diagnosing IE issues:

http://home.nyc.rr.com/itandetnik/passthruapp.zip

[DH]

Swede wrote:

So, what does PHP have to do with this? Well, my download links to
purchaseware are encrypted by the PHP "ewportal":
http://www.eliteweaver.co.uk/antifra....php?page=Home

Try using WinZip or similar application to make the ZIP files and see if
the problem goes away. I've been using eliteweaver for almost 2 years,
problem free (aside from some really ugly code), although my downloads
were all compiled using WinZip.

[Swede]

Thanks! I'll have a look at it! /Lars

ch***********@hotmail.com wrote:

Don't know if it would help in this case, but Igor Tandetnik's
PassThruAPP is very useful for diagnosing IE issues:

http://home.nyc.rr.com/itandetnik/passthruapp.zip

[Swede]

DH wrote:

Swede wrote:

So, what does PHP have to do with this? Well, my download links to
purchaseware are encrypted by the PHP "ewportal":
http://www.eliteweaver.co.uk/antifra....php?page=Home

Try using WinZip or similar application to make the ZIP files and see if
the problem goes away. I've been using eliteweaver for almost 2 years,
problem free (aside from some really ugly code), although my downloads
were all compiled using WinZip.

Thanks for the tip! Unfortunately this is how I have done it all the
time. /Lars