471,354 Members | 1,520 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,354 software developers and data experts.

zipfile 2GB problems?

Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a thread
http://mail.python.org/pipermail/pyt...il/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?

Jul 15 '07 #1
12 4411
Jim
On Jul 14, 10:35 pm, xamdam <maxkhe...@gmail.comwrote:
Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a threadhttp://mail.python.org/pipermail/python-dev/2005-April/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?
Hi,
I use PKZIP V9.00 and maintain a 2.5GB zipfile that consists of over
6,600 zipped files.
There is no problem with reading or writing.
Thanks,
Jim

Jul 15 '07 #2
Jim wrote:
On Jul 14, 10:35 pm, xamdam <maxkhe...@gmail.comwrote:
>Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a threadhttp://mail.python.org/pipermail/python-dev/2005-April/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?

Hi,
I use PKZIP V9.00 and maintain a 2.5GB zipfile that consists of over
6,600 zipped files.
There is no problem with reading or writing.
.... with reading or writing *using the Python libraries*?

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Jul 15 '07 #3
On Jul 15, 12:35 am, xamdam <maxkhe...@gmail.comwrote:
Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a thread http://mail.python.org/pipermail/pyt...il/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?
Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
Jul 15 '07 #4
Hi,
I use PKZIP V9.00 and maintain a 2.5GB zipfile that consists of over
6,600 zipped files.
There is no problem with reading or writing.
Thanks,
Jim
Which version of Python? What platform?

thanks,
max

Jul 15 '07 #5
xamdam wrote:
On Jul 15, 12:35 am, xamdam <maxkhe...@gmail.comwrote:
>Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a thread http://mail.python.org/pipermail/pyt...il/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?

Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.

It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Jul 15 '07 #6
>
Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.

It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.

Jul 15 '07 #7
xamdam wrote:
>>Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?

Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.
Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Jul 15 '07 #8
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
>Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.

Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.
It's failing in the ZipFile constructor, which ends up throwing

zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"

Jul 16 '07 #9
xamdam wrote:
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
>xamdam wrote:
>>>>Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.
Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.

It's failing in the ZipFile constructor, which ends up throwing

zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"
Well, that one bald line by itself certainly doesn't give *me* any
traction an what the problem might be. Getting information about this
problem is like (i.e. as difficult as) pulling teeth!

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Jul 16 '07 #10
On Jul 16, 3:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.
Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.
It's failing in the ZipFile constructor, which ends up throwing
zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"

Well, that one bald line by itself certainly doesn't give *me* any
traction an what the problem might be. Getting information about this
problem is like (i.e. as difficult as) pulling teeth!

regards
Steve
Well, Steve, I don't know what else to tell you - I certainly do not
want to cause you tooth ache! I was sort of hoping for some ZIP expert
to go, 'oh, yeah'...

Jul 16 '07 #11
On Jul 17, 4:59 am, xamdam <maxkhe...@gmail.comwrote:
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
>>Additional info: my file is from a data provider, do not know what
>>they used to compress it. Previous files worked ok, this one is the
>>1st over 2GB. Winzip has no problem with it.
>It could be you are using a Python with an in-built limit of 2GB on file
>size. What happens if you open the file, seek past the 2GB point and
>then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.
Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.

It's failing in the ZipFile constructor, which ends up throwing

zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"
Suggestion 1: RTEDManual

Manual says:
"""
12.4.1 ZipFile Objects

class ZipFile( file[, mode[, compression[, allowZip64]]])
[snip]
If allowZip64 is True zipfile will create ZIP files that use the ZIP64
extensions when the zipfile is larger than 2 GB. If it is false (the
default) zipfile will raise an exception when the ZIP file would
require ZIP64 extensions. ZIP64 extensions are disabled by default
because the default zip and unzip commands on Unix (the InfoZIP
utilities) don't support these extensions.
"""

It's vague about whether allowZip64=True is needed for reading -- have
you tried it?

Suggestion 2: RTEDCode. At the line you quoted, it is checking the
first 4 bytes of the file for a signature: 'PK\x03\x04'
Do this:
print repr(open('yourfile.zip', 'rb').read(4))
Tell us what you find.

Suggestion 3: RTEDWordsOfWisdomOfSteveHolden. IOW supply the *WHOLE*
EDing traceback.

HTH,
John

Jul 16 '07 #12
On Jul 17, 7:37 am, xamdam <maxkhe...@gmail.comwrote:
On Jul 16, 3:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
>xamdam wrote:
>>>>Additional info: my file is from a data provider, do not know what
>>>>they used to compress it. Previous files worked ok, this one is the
>>>>1st over 2GB. Winzip has no problem with it.
>>>It could be you are using a Python with an in-built limit of 2GB on file
>>>size. What happens if you open the file, seek past the 2GB point and
>>>then try and read data?
>>Steve,
>>I used is_zipfile function, which from does exactly that from a quick
>>read - goes to the end, reads expected string. It worked fine. Tried
>>regular Windows 2.4.4 and 2.5.1, same result.
>Fine, if it isn't file size limitations I suspect you need to post some
>code and a traceback so we can get better insight into exactly what the
>problem is.
It's failing in the ZipFile constructor, which ends up throwing
zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"
Well, that one bald line by itself certainly doesn't give *me* any
traction an what the problem might be. Getting information about this
problem is like (i.e. as difficult as) pulling teeth!
regards
Steve

Well, Steve, I don't know what else to tell you
"post some code and a traceback"
I certainly do not
want to cause you tooth ache!
I suspect that you are inducing unwanted symptoms at the *other* end
of Steve's alimentary canal.
I was sort of hoping for some ZIP expert
to go, 'oh, yeah'...
Hope all you want, but supply the information you are asked for.

Jul 16 '07 #13

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by LC | last post: by
19 posts views Thread by Gerson Kurz | last post: by
1 post views Thread by Waitman Gobble | last post: by
5 posts views Thread by Waguy | last post: by
3 posts views Thread by towers | last post: by
5 posts views Thread by Neil Crighton | last post: by
1 post views Thread by John Machin | last post: by
reply views Thread by XIAOLAOHU | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.