473,218 Members | 1,808 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,218 software developers and data experts.

zipfile 2GB problems?

Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a thread
http://mail.python.org/pipermail/pyt...il/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?

Jul 15 '07 #1
12 4606
Jim
On Jul 14, 10:35 pm, xamdam <maxkhe...@gmail.comwrote:
Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a threadhttp://mail.python.org/pipermail/python-dev/2005-April/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?
Hi,
I use PKZIP V9.00 and maintain a 2.5GB zipfile that consists of over
6,600 zipped files.
There is no problem with reading or writing.
Thanks,
Jim

Jul 15 '07 #2
Jim wrote:
On Jul 14, 10:35 pm, xamdam <maxkhe...@gmail.comwrote:
>Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a threadhttp://mail.python.org/pipermail/python-dev/2005-April/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?

Hi,
I use PKZIP V9.00 and maintain a 2.5GB zipfile that consists of over
6,600 zipped files.
There is no problem with reading or writing.
.... with reading or writing *using the Python libraries*?

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Jul 15 '07 #3
On Jul 15, 12:35 am, xamdam <maxkhe...@gmail.comwrote:
Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a thread http://mail.python.org/pipermail/pyt...il/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?
Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
Jul 15 '07 #4
Hi,
I use PKZIP V9.00 and maintain a 2.5GB zipfile that consists of over
6,600 zipped files.
There is no problem with reading or writing.
Thanks,
Jim
Which version of Python? What platform?

thanks,
max

Jul 15 '07 #5
xamdam wrote:
On Jul 15, 12:35 am, xamdam <maxkhe...@gmail.comwrote:
>Hi fellas,
I am experiencing problems reading a 2GB zipfile consisting of
multiple zipped files. I found a thread http://mail.python.org/pipermail/pyt...il/053027.html
that mentions a problem on the writing side, does such a problem exist
on a reading side? I am using 2.4.1, perhaps there is a fix in a later
version?

Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.

It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Jul 15 '07 #6
>
Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.

It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.

Jul 15 '07 #7
xamdam wrote:
>>Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?

Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.
Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Jul 15 '07 #8
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
>Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.

Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.
It's failing in the ZipFile constructor, which ends up throwing

zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"

Jul 16 '07 #9
xamdam wrote:
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
>xamdam wrote:
>>>>Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.
Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.

It's failing in the ZipFile constructor, which ends up throwing

zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"
Well, that one bald line by itself certainly doesn't give *me* any
traction an what the problem might be. Getting information about this
problem is like (i.e. as difficult as) pulling teeth!

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

Jul 16 '07 #10
On Jul 16, 3:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
Additional info: my file is from a data provider, do not know what
they used to compress it. Previous files worked ok, this one is the
1st over 2GB. Winzip has no problem with it.
It could be you are using a Python with an in-built limit of 2GB on file
size. What happens if you open the file, seek past the 2GB point and
then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.
Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.
It's failing in the ZipFile constructor, which ends up throwing
zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"

Well, that one bald line by itself certainly doesn't give *me* any
traction an what the problem might be. Getting information about this
problem is like (i.e. as difficult as) pulling teeth!

regards
Steve
Well, Steve, I don't know what else to tell you - I certainly do not
want to cause you tooth ache! I was sort of hoping for some ZIP expert
to go, 'oh, yeah'...

Jul 16 '07 #11
On Jul 17, 4:59 am, xamdam <maxkhe...@gmail.comwrote:
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
>>Additional info: my file is from a data provider, do not know what
>>they used to compress it. Previous files worked ok, this one is the
>>1st over 2GB. Winzip has no problem with it.
>It could be you are using a Python with an in-built limit of 2GB on file
>size. What happens if you open the file, seek past the 2GB point and
>then try and read data?
Steve,
I used is_zipfile function, which from does exactly that from a quick
read - goes to the end, reads expected string. It worked fine. Tried
regular Windows 2.4.4 and 2.5.1, same result.
Fine, if it isn't file size limitations I suspect you need to post some
code and a traceback so we can get better insight into exactly what the
problem is.

It's failing in the ZipFile constructor, which ends up throwing

zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"
Suggestion 1: RTEDManual

Manual says:
"""
12.4.1 ZipFile Objects

class ZipFile( file[, mode[, compression[, allowZip64]]])
[snip]
If allowZip64 is True zipfile will create ZIP files that use the ZIP64
extensions when the zipfile is larger than 2 GB. If it is false (the
default) zipfile will raise an exception when the ZIP file would
require ZIP64 extensions. ZIP64 extensions are disabled by default
because the default zip and unzip commands on Unix (the InfoZIP
utilities) don't support these extensions.
"""

It's vague about whether allowZip64=True is needed for reading -- have
you tried it?

Suggestion 2: RTEDCode. At the line you quoted, it is checking the
first 4 bytes of the file for a signature: 'PK\x03\x04'
Do this:
print repr(open('yourfile.zip', 'rb').read(4))
Tell us what you find.

Suggestion 3: RTEDWordsOfWisdomOfSteveHolden. IOW supply the *WHOLE*
EDing traceback.

HTH,
John

Jul 16 '07 #12
On Jul 17, 7:37 am, xamdam <maxkhe...@gmail.comwrote:
On Jul 16, 3:39 pm, Steve Holden <st...@holdenweb.comwrote:
xamdam wrote:
On Jul 15, 5:39 pm, Steve Holden <st...@holdenweb.comwrote:
>xamdam wrote:
>>>>Additional info: my file is from a data provider, do not know what
>>>>they used to compress it. Previous files worked ok, this one is the
>>>>1st over 2GB. Winzip has no problem with it.
>>>It could be you are using a Python with an in-built limit of 2GB on file
>>>size. What happens if you open the file, seek past the 2GB point and
>>>then try and read data?
>>Steve,
>>I used is_zipfile function, which from does exactly that from a quick
>>read - goes to the end, reads expected string. It worked fine. Tried
>>regular Windows 2.4.4 and 2.5.1, same result.
>Fine, if it isn't file size limitations I suspect you need to post some
>code and a traceback so we can get better insight into exactly what the
>problem is.
It's failing in the ZipFile constructor, which ends up throwing
zipfile(2.4.3).py:291raise BadZipfile, "Bad magic number for file
header"
Well, that one bald line by itself certainly doesn't give *me* any
traction an what the problem might be. Getting information about this
problem is like (i.e. as difficult as) pulling teeth!
regards
Steve

Well, Steve, I don't know what else to tell you
"post some code and a traceback"
I certainly do not
want to cause you tooth ache!
I suspect that you are inducing unwanted symptoms at the *other* end
of Steve's alimentary canal.
I was sort of hoping for some ZIP expert
to go, 'oh, yeah'...
Hope all you want, but supply the information you are asked for.

Jul 16 '07 #13

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: LC | last post by:
Hi, I'm having a problem using the zipfile module in Windows 2000 sp4. When I use it to zip a small file it works fine, but large file doesnt. Here's the error msg i get......
19
by: Gerson Kurz | last post by:
AAAAAAAARG I hate the way python handles unicode. Here is a nice problem for y'all to enjoy: say you have a variable thats unicode directory = u"c:\temp" Its unicode not because you want it...
4
by: vincent_delft | last post by:
I've a simple python script that read a directory and put the files into a Zip file. I'm using the os.walk method to get the directory content, I'm creating ZipInfo objects and set "filename",...
1
by: Waitman Gobble | last post by:
Hello, I am new to Python. I am having trouble with zipfile.py. On a Linux machine with python 2.4.2 I have trouble opening a zipfile. Python is complaining about the bit where it does a...
5
by: Waguy | last post by:
Hi all, I am new to python and want to create a process to unzip large numbers of zip files I get from a SOAP application. The files all have a ZIP extention and can be unzipped using WinZip. ...
3
bvdet
by: bvdet | last post by:
Following is an example that may provide a solution to you: """ Function makeArchive is a wrapper for the Python class zipfile.ZipFile 'fileList' is a list of file names - full path each name...
3
by: towers | last post by:
Hi I'm probably doing something stupid but I've run into a problem whereby I'm trying to add a csv file to a zip archive - see example code below. The csv just has several rows with carriage...
5
by: Neil Crighton | last post by:
I'm using the zipfile library to read a zip file in Windows, and it seems to be adding too many newlines to extracted files. I've found that for extracted text-encoded files, removing all instances...
1
by: John Machin | last post by:
On Jun 4, 8:06 pm, jwesonga <crazylun...@gmail.comwrote: Nothing is ever as it seems. Let's try to work backwards from the error message ... and we don't need your magnificent script, just the...
1
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
0
by: VivesProcSPL | last post by:
Obviously, one of the original purposes of SQL is to make data query processing easy. The language uses many English-like terms and syntax in an effort to make it easy to learn, particularly for...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 3 Jan 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). For other local times, please check World Time Buddy In...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
0
by: mar23 | last post by:
Here's the situation. I have a form called frmDiceInventory with subform called subfrmDice. The subform's control source is linked to a query called qryDiceInventory. I've been trying to pick up the...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.