how can I put a 1Gb file in a zipfile??

[Bennie]

Hi,

I have a problem with ZipFile.
It works okay untily I come across a file that is greater then 1Gb.
Then it exit with the error:
OverflowError: long int too large to convert to int

How can I fix this?

souce:
zip = zipfile.ZipFile(file, 'w')

for all in os.walk(os.getcwd()):
path = all[0]
for document in all[2]:
zipaccview.write(path + os.sep + document)

zip.close()

Bennie

[Christos TZOTZIOY Georgiou]

On Sun, 20 Mar 2005 10:44:06 +0100, rumours say that Bennie <ro*****@wanadoo.nl>
might have written:

Hi,

I have a problem with ZipFile.
It works okay untily I come across a file that is greater then 1Gb.
Then it exit with the error:
OverflowError: long int too large to convert to int

How can I fix this?
AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip
and to the total size of the zip file. This limit comes from the zip file
specification (32 bit offsets).
souce:
zip = zipfile.ZipFile(file, 'w')

for all in os.walk(os.getcwd()):
path = all[0]
for document in all[2]:
zipaccview.write(path + os.sep + document)

zip.close()

Can it be that you are creating a zip file that its total size exceeds the
limit?
--
TZOTZIOY, I speak England very best.
"Be strict when sending and tolerant when receiving." (from RFC1958)
I really should keep that in mind when talking with people, actually...

[bennie]

Christos TZOTZIOY Georgiou wrote:

On Sun, 20 Mar 2005 10:44:06 +0100, rumours say that Bennie <ro*****@wanadoo.nl>
might have written:

Hi,

I have a problem with ZipFile.
It works okay untily I come across a file that is greater then 1Gb.
Then it exit with the error:
OverflowError: long int too large to convert to int

How can I fix this?

AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip
and to the total size of the zip file. This limit comes from the zip file
specification (32 bit offsets).

souce:
zip = zipfile.ZipFile(file, 'w')

for all in os.walk(os.getcwd()):
path = all[0]
for document in all[2]:
zipaccview.write(path + os.sep + document)

zip.close()

Can it be that you are creating a zip file that its total size exceeds the
limit?

That is possible.
But with Winzip program it can.
How come that is with ZipFile not works

[Jeff Epler]

The limits of ZIP files according to the folks who make info-zip:
http://www.info-zip.org/pub/infozip/FAQ.html#limits

statistic limit
number of files 65,536
uncompressed size of a single file 4 GB
compressed size of a single file 4 GB
total size of archive 256 TB
maximum path/filename length 64 KB

I had no trouble creating a zip file from a 4GB file filled with '\0'
bytes:
$ python bennie.py
$ ls -ls test.zip big
12 -rw-rw-r-- 1 jepler jepler 4294967296 Mar 20 14:11 big
4084 -rw-rw-r-- 1 jepler jepler 4174545 Mar 20 14:14 test.zip

I'm using Python 2.3.3 on Fedora Core 2.
#------------------------------------------------------------------------
# bennie.py
def make_4gb_file(f):
f = open(f, "w")
f.seek ( 4 * 1024 * 1024 * 1024 - 1)
f.write("\0")
f.close()

import zipfile
z = zipfile.ZipFile("/tmp/test.zip", "w", zipfile.ZIP_DEFLATED)
make_4gb_file("/tmp/big")
z.write("/tmp/big")
z.close()
#------------------------------------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCPdpOJd01MZaTXX0RAngaAJ9tkc4tCUAwJmlio7/9Pn46Dyh9bgCfdRQW
4rQqWnqLljuDUH/d6NBfoeM=
=XSAF
-----END PGP SIGNATURE-----

[Bengt Richter]

On Sun, 20 Mar 2005 14:17:20 -0600, Jeff Epler <je****@unpythonic.net> wrote:

--liOOAslEiF7prFVr
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

The limits of ZIP files according to the folks who make info-zip:
http://www.info-zip.org/pub/infozip/FAQ.html#limits

statistic limit
number of files 65,536
uncompressed size of a single file 4 GB
compressed size of a single file 4 GB
total size of archive 256 TB
maximum path/filename length 64 KB=20

I had no trouble creating a zip file from a 4GB file filled with '\0'
bytes:
$ python bennie.py
$ ls -ls test.zip big
12 -rw-rw-r-- 1 jepler jepler 4294967296 Mar 20 14:11 big
4084 -rw-rw-r-- 1 jepler jepler 4174545 Mar 20 14:14 test.zip

I'm using Python 2.3.3 on Fedora Core 2.
#------------------------------------------------------------------------
# bennie.py
def make_4gb_file(f):
f =3D open(f, "w")
f.seek ( 4 * 1024 * 1024 * 1024 - 1)
f.write("\0")
f.close()
Not quite OT[?]:
This makes me think there ought to be a way of making at least python's builtin open see virtual file system objects,
analogous to StringIO creating file objects.

If we had a mountvfs('/some/unix/and/or/maybe/win/style/path/vfs', vfsclass(some, args, if_desired))
which would result in that open(/some/unix/and/or/maybe/win/style/path/vfs/morepath/filename.ext', mode)
(where vfsclass(some, args, if_desired) => vsfclass_intance) would call
vfsclass_instance.open('morepath.filename.ext', mode) which could then return an object that could support
file operations like returning 4gb of virtually read-by-read-method data, or otherwise acting like an open file
object of a real file system that python programs and library function using open and file would find
if given the mounted path. Subdirectories could be fixed for starters, but virtualizing subdirectory creation
etc would be possible if you intercepted the right interface calls and implemented it in the vfs.

This would let you define a virtual file in place of the real file above, and also would allow a lot of transparent
testing of file-using software that takes paths and file names, not open file objects.

Of course you can't affect what the underlying os sees without getting into its file system machinery, but
being able to mount virtual file systems into what os.open sees would cover a lot of ground ISTM. One
could argue pro and con about supporting virtual mounts into both unix and windows-style paths.
import zipfile
z =3D zipfile.ZipFile("/tmp/test.zip", "w", zipfile.ZIP_DEFLATED)
make_4gb_file("/tmp/big")
z.write("/tmp/big")
z.close()
#------------------------------------------------------------------------

Regards,
Bengt Richter

[RM]

>From what I understand, Winzip uses their own proprietary version of a
Zip format. That means that you will only be able open those archives
with Winzip. (IOW, you are locked in)

I think someone mentioned having developed a Python module for the 7zip
fomat, but I may be wrong. In any case, you could simply write a
wrapper over the command line version of 7zip and your problem would be
solved. There is a Linux version as well.
(http://p7zip.sourceforge.net/)
bennie wrote:

Christos TZOTZIOY Georgiou wrote:

AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip and to the total size of the zip file. This limit comes from the zip file specification (32 bit offsets).
Can it be that you are creating a zip file that its total size exceeds the limit?

That is possible.
But with Winzip program it can.
How come that is with ZipFile not works

[Martin v. Löwis]

Bennie wrote:

How can I fix this?

Since this is the question that you actually asked, there
is an easy answer. Investigate the problem in detail, understand
the source code of the zipfile module, and the format of zip files,
and develop a change to the code to correct the behaviour.
Then submit a patch to sf.net/projects/python.

Regards,
Martin