Corrupted ZIP files during download via PHP

Hello,

I am posting this rather lengthy description here in hope of getting at
least some insight into a problem that is really hurting my business,
since I have come to the conclusion that my problem is at least partly
PHP-related.

Some time now I have been having transfer problems, i.e. my customers
downloads of large binary (ZIP) files are often corrupt.

A corrupted file now and then can of course never be avoided, but I get
several complaints from customers every day. Of any specific file,
approx. 1 out of 4 downloads (larger than 100kB or so) are affected.
I have spent months trying to find the source of the problems without
success, but I have been able to rule out many possible causes. Here are
some facts:

1) The actual files are OK on the server. Direct (unencrypted) links are
fine, so it's not a general problem.

2) About 50% of the customers download (the exact same files) from
another source. I don't even get 1 complaint every two months from those
customers, while I get 1-3 complaints _every day_ from those who
download from my server.

3) The customers have various SW/HW/OS, so it's very unlikely that it's
a user problem. Plus, the same customer often download from both places,
but only have problems with the files from my server.

4) I have shut down all scripts for a while, so that only a static
website remained, but it didn't help.

5) I have had this problem on three servers: The present RaQ550, the
previous RaQ4 and the RaQ4 I had before that. So it's highly unlikely
that it is server-related.

6) I can easily reproduce the problem myself. As I looked into this I
found a systematic error:

Analyzing corrupt vs. clean ZIP file downloads with a binary/hex editor
clearly shows that the error is systematic. Clean ZIP files have a lot
of '00h' bytes in them. In ALL of the corrupt ZIP files ALL (well, most)
of these bytes have been exchanged for '5C30h', i.e. TWO bytes ( 00h ->
5C30h ) !

Since '5C30h' equals an escaped zero, i.e. "\0", my guess is that there
is a translation error somewhere.

So, what does PHP have to do with this? Well, my download links to
purchaseware are encrypted by the PHP "ewportal":
http://www.eliteweaver.co.uk/antifra....php?page=Home

From the few facts I have been able to uncover it seems there is
sometimes a problem with corrupted ZIP files depending on the
setting/compilation of ZLIB. If anyone on this forum have experienced
similar things I would very much appreciate if they could share.
My servers PHP configuration is at:
http://www.snith.com/phptest.php

Best regards, Lars

re: Corrupted ZIP files during download via PHP

*** Swede escribió/wrote (Tue, 15 Feb 2005 21:16:18 +0100):[color=blue]
> Some time now I have been having transfer problems, i.e. my customers
> downloads of large binary (ZIP) files are often corrupt.[/color]

I suggest that you check the server's MIME type for *.zip files.

[color=blue]
> From the few facts I have been able to uncover it seems there is
> sometimes a problem with corrupted ZIP files depending on the
> setting/compilation of ZLIB.[/color]

Do you compress the files on the fly?


--
-+ Álvaro G. Vicario - Burgos, Spain
+- http://www.demogracia.com (la web de humor barnizada para la intemperie)
++ Manda tus dudas al grupo, no a mi buzón
-+ Send your questions to the group, not to my mailbox
--

re: Corrupted ZIP files during download via PHP

Hello Alvaro,

Alvaro G. Vicario wrote:
[color=blue]
> *** Swede escribió/wrote (Tue, 15 Feb 2005 21:16:18 +0100):
>[color=green]
>>Some time now I have been having transfer problems, i.e. my customers
>>downloads of large binary (ZIP) files are often corrupt.[/color]
>
> I suggest that you check the server's MIME type for *.zip files.[/color]

I have already tried that, it was set to some non-descript application.
I changed it to "octet-stream" together with "*.exe" and the other
binaries, but it did not help. Is there any more place than "mime.types"
where I could change this?

I have seen some brief comments about "commenting out the header which
gets the mimetype of the file in class.download.inc", but I have no idea
how/where to do that, and if it is related to my problem.
[color=blue][color=green]
>> From the few facts I have been able to uncover it seems there is
>>sometimes a problem with corrupted ZIP files depending on the
>>setting/compilation of ZLIB.[/color]
>
> Do you compress the files on the fly?[/color]

Not intentionally...they are already ZIPs on the server disk.

Thanks for your input so far, I think you are definitely on the right
path! Any more good ideas?


Best regards, Lars

re: Corrupted ZIP files during download via PHP

*** Swede escribió/wrote (Tue, 15 Feb 2005 21:52:39 +0100):[color=blue]
> I have already tried that, it was set to some non-descript application.
> I changed it to "octet-stream" together with "*.exe" and the other
> binaries, but it did not help. Is there any more place than "mime.types"
> where I could change this?[/color]

My suggestion only came because maybe the server was using the default type
(usually text/html) and was messing with character encoding or God knows
what.

Anyway, since you seem to be using a PHP script to serve files then it's
the script the one who should be setting the MIME type; otherwise you'll
get the default type. Something similar to this should do:

header('Content-Type: application/zip');

Internet Explorer is known for having problems with PDF files (esp. if you
use sessions) but I'm not sure about ZIPs.



--
-+ Álvaro G. Vicario - Burgos, Spain
+- http://www.demogracia.com (la web de humor barnizada para la intemperie)
++ Manda tus dudas al grupo, no a mi buzón
-+ Send your questions to the group, not to my mailbox
--

re: Corrupted ZIP files during download via PHP

Hello Again,

Alvaro G. Vicario wrote:
[color=blue]
> *** Swede escribió/wrote (Tue, 15 Feb 2005 21:52:39 +0100):
>[color=green]
>>I have already tried that, it was set to some non-descript application.
>>I changed it to "octet-stream" together with "*.exe" and the other
>>binaries, but it did not help. Is there any more place than "mime.types"
>>where I could change this?[/color]
>
> My suggestion only came because maybe the server was using the default type
> (usually text/html) and was messing with character encoding or God knows
> what.
>
> Anyway, since you seem to be using a PHP script to serve files then it's
> the script the one who should be setting the MIME type; otherwise you'll
> get the default type. Something similar to this should do:
>
> header('Content-Type: application/zip');[/color]

Sounds like a good idea, should this go into "php.ini", or in the script
itself ?
[color=blue]
>
> Internet Explorer is known for having problems with PDF files (esp. if you
> use sessions) but I'm not sure about ZIPs.[/color]

Yes, IE causes a lot of problems, but I see this also on my Mozilla/Firefox.


Best regards, Lars

re: Corrupted ZIP files during download via PHP

The conversion of binary zeros to "\0"s is consistent with
magic_quotes_runtime = On.

re: Corrupted ZIP files during download via PHP

*** Swede escribió/wrote (Tue, 15 Feb 2005 22:15:08 +0100):[color=blue][color=green]
>> header('Content-Type: application/zip');[/color]
>
> Sounds like a good idea, should this go into "php.ini", or in the script
> itself ?[/color]

Maybe it's a good idea not to mess with this if you are unsure of what it
does. This is just PHP code, not a configuration directive. How to use it
depends on how your script is coded.
[color=blue][color=green]
>> Internet Explorer is known for having problems with PDF files (esp. if you
>> use sessions) but I'm not sure about ZIPs.[/color]
>
> Yes, IE causes a lot of problems, but I see this also on my Mozilla/Firefox.[/color]

If you use Firefox, please install the "Live HTTP Headers" extension and
find out yourself what MIME type is used.



--
-+ Álvaro G. Vicario - Burgos, Spain
+- http://www.demogracia.com (la web de humor barnizada para la intemperie)
++ Manda tus dudas al grupo, no a mi buzón
-+ Send your questions to the group, not to my mailbox
--

re: Corrupted ZIP files during download via PHP

Hello,

Thanks for the tip. Unfortunately all of magic_quotes_gpc,
magic_quotes_runtime, magic_quotes_sybase are "off" in the php.ini, so
this should not be the cause as far as I can understand.

Best regards, Lars

chernyshevsky@hotmail.com wrote:
[color=blue]
> The conversion of binary zeros to "\0"s is consistent with
> magic_quotes_runtime = On.
>[/color]

re: Corrupted ZIP files during download via PHP

Hello Alvaro,

Thanks for your input, I really appreciate it!
Unfortunately the script is not my own to start with, I do many
languages but have not got around to PHP yet. I will try it out
(reversible...). But it will have to be tomorrow, it's bedtime for me
here now (Norway)!

Best regards, Lars


Alvaro G. Vicario wrote:
[color=blue]
> *** Swede escribió/wrote (Tue, 15 Feb 2005 22:15:08 +0100):
>[color=green][color=darkred]
>>>header('Content-Type: application/zip');[/color]
>>
>>Sounds like a good idea, should this go into "php.ini", or in the script
>>itself ?[/color]
>
>
> Maybe it's a good idea not to mess with this if you are unsure of what it
> does. This is just PHP code, not a configuration directive. How to use it
> depends on how your script is coded.
>
>[color=green][color=darkred]
>>>Internet Explorer is known for having problems with PDF files (esp. if you
>>>use sessions) but I'm not sure about ZIPs.[/color]
>>
>>Yes, IE causes a lot of problems, but I see this also on my Mozilla/Firefox.[/color]
>
>
> If you use Firefox, please install the "Live HTTP Headers" extension and
> find out yourself what MIME type is used.
>
>
>[/color]

re: Corrupted ZIP files during download via PHP

Don't know if it would help in this case, but Igor Tandetnik's
PassThruAPP is very useful for diagnosing IE issues:

http://home.nyc.rr.com/itandetnik/passthruapp.zip

re: Corrupted ZIP files during download via PHP

Swede wrote:
[color=blue]
> So, what does PHP have to do with this? Well, my download links to
> purchaseware are encrypted by the PHP "ewportal":
> http://www.eliteweaver.co.uk/antifra....php?page=Home[/color]

Try using WinZip or similar application to make the ZIP files and see if
the problem goes away. I've been using eliteweaver for almost 2 years,
problem free (aside from some really ugly code), although my downloads
were all compiled using WinZip.

re: Corrupted ZIP files during download via PHP

Thanks! I'll have a look at it! /Lars

chernyshevsky@hotmail.com wrote:
[color=blue]
> Don't know if it would help in this case, but Igor Tandetnik's
> PassThruAPP is very useful for diagnosing IE issues:
>
> http://home.nyc.rr.com/itandetnik/passthruapp.zip
>[/color]

re: Corrupted ZIP files during download via PHP

DH wrote:
[color=blue]
> Swede wrote:
>[color=green]
>> So, what does PHP have to do with this? Well, my download links to
>> purchaseware are encrypted by the PHP "ewportal":
>> http://www.eliteweaver.co.uk/antifra....php?page=Home[/color]
>
>
> Try using WinZip or similar application to make the ZIP files and see if
> the problem goes away. I've been using eliteweaver for almost 2 years,
> problem free (aside from some really ugly code), although my downloads
> were all compiled using WinZip.[/color]

Thanks for the tip! Unfortunately this is how I have done it all the
time. /Lars