473,543 Members | 1,983 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

how can I put a 1Gb file in a zipfile??

Hi,

I have a problem with ZipFile.
It works okay untily I come across a file that is greater then 1Gb.
Then it exit with the error:
OverflowError: long int too large to convert to int

How can I fix this?

souce:
zip = zipfile.ZipFile (file, 'w')

for all in os.walk(os.getc wd()):
path = all[0]
for document in all[2]:
zipaccview.writ e(path + os.sep + document)

zip.close()

Bennie
Jul 18 '05 #1
6 4159
On Sun, 20 Mar 2005 10:44:06 +0100, rumours say that Bennie <ro*****@wanado o.nl>
might have written:
Hi,

I have a problem with ZipFile.
It works okay untily I come across a file that is greater then 1Gb.
Then it exit with the error:
OverflowError: long int too large to convert to int

How can I fix this?
AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip
and to the total size of the zip file. This limit comes from the zip file
specification (32 bit offsets).
souce:
zip = zipfile.ZipFile (file, 'w')

for all in os.walk(os.getc wd()):
path = all[0]
for document in all[2]:
zipaccview.writ e(path + os.sep + document)

zip.close()


Can it be that you are creating a zip file that its total size exceeds the
limit?
--
TZOTZIOY, I speak England very best.
"Be strict when sending and tolerant when receiving." (from RFC1958)
I really should keep that in mind when talking with people, actually...
Jul 18 '05 #2
Christos TZOTZIOY Georgiou wrote:
On Sun, 20 Mar 2005 10:44:06 +0100, rumours say that Bennie <ro*****@wanado o.nl>
might have written:

Hi,

I have a problem with ZipFile.
It works okay untily I come across a file that is greater then 1Gb.
Then it exit with the error:
OverflowError: long int too large to convert to int

How can I fix this?

AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip
and to the total size of the zip file. This limit comes from the zip file
specification (32 bit offsets).

souce:
zip = zipfile.ZipFile (file, 'w')

for all in os.walk(os.getc wd()):
path = all[0]
for document in all[2]:
zipaccview.writ e(path + os.sep + document)

zip.close()

Can it be that you are creating a zip file that its total size exceeds the
limit?

That is possible.
But with Winzip program it can.
How come that is with ZipFile not works
Jul 18 '05 #3
The limits of ZIP files according to the folks who make info-zip:
http://www.info-zip.org/pub/infozip/FAQ.html#limits

statistic limit
number of files 65,536
uncompressed size of a single file 4 GB
compressed size of a single file 4 GB
total size of archive 256 TB
maximum path/filename length 64 KB

I had no trouble creating a zip file from a 4GB file filled with '\0'
bytes:
$ python bennie.py
$ ls -ls test.zip big
12 -rw-rw-r-- 1 jepler jepler 4294967296 Mar 20 14:11 big
4084 -rw-rw-r-- 1 jepler jepler 4174545 Mar 20 14:14 test.zip

I'm using Python 2.3.3 on Fedora Core 2.
#------------------------------------------------------------------------
# bennie.py
def make_4gb_file(f ):
f = open(f, "w")
f.seek ( 4 * 1024 * 1024 * 1024 - 1)
f.write("\0")
f.close()

import zipfile
z = zipfile.ZipFile ("/tmp/test.zip", "w", zipfile.ZIP_DEF LATED)
make_4gb_file("/tmp/big")
z.write("/tmp/big")
z.close()
#------------------------------------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCPdpOJd0 1MZaTXX0RAngaAJ 9tkc4tCUAwJmlio 7/9Pn46Dyh9bgCfdR QW
4rQqWnqLljuDUH/d6NBfoeM=
=XSAF
-----END PGP SIGNATURE-----

Jul 18 '05 #4
On Sun, 20 Mar 2005 14:17:20 -0600, Jeff Epler <je****@unpytho nic.net> wrote:

--liOOAslEiF7prFV r
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

The limits of ZIP files according to the folks who make info-zip:
http://www.info-zip.org/pub/infozip/FAQ.html#limits

statistic limit
number of files 65,536
uncompressed size of a single file 4 GB
compressed size of a single file 4 GB
total size of archive 256 TB
maximum path/filename length 64 KB=20

I had no trouble creating a zip file from a 4GB file filled with '\0'
bytes:
$ python bennie.py
$ ls -ls test.zip big
12 -rw-rw-r-- 1 jepler jepler 4294967296 Mar 20 14:11 big
4084 -rw-rw-r-- 1 jepler jepler 4174545 Mar 20 14:14 test.zip

I'm using Python 2.3.3 on Fedora Core 2.
#------------------------------------------------------------------------
# bennie.py
def make_4gb_file(f ):
f =3D open(f, "w")
f.seek ( 4 * 1024 * 1024 * 1024 - 1)
f.write("\0")
f.close()
Not quite OT[?]:
This makes me think there ought to be a way of making at least python's builtin open see virtual file system objects,
analogous to StringIO creating file objects.

If we had a mountvfs('/some/unix/and/or/maybe/win/style/path/vfs', vfsclass(some, args, if_desired))
which would result in that open(/some/unix/and/or/maybe/win/style/path/vfs/morepath/filename.ext', mode)
(where vfsclass(some, args, if_desired) => vsfclass_intanc e) would call
vfsclass_instan ce.open('morepa th.filename.ext ', mode) which could then return an object that could support
file operations like returning 4gb of virtually read-by-read-method data, or otherwise acting like an open file
object of a real file system that python programs and library function using open and file would find
if given the mounted path. Subdirectories could be fixed for starters, but virtualizing subdirectory creation
etc would be possible if you intercepted the right interface calls and implemented it in the vfs.

This would let you define a virtual file in place of the real file above, and also would allow a lot of transparent
testing of file-using software that takes paths and file names, not open file objects.

Of course you can't affect what the underlying os sees without getting into its file system machinery, but
being able to mount virtual file systems into what os.open sees would cover a lot of ground ISTM. One
could argue pro and con about supporting virtual mounts into both unix and windows-style paths.
import zipfile
z =3D zipfile.ZipFile ("/tmp/test.zip", "w", zipfile.ZIP_DEF LATED)
make_4gb_file( "/tmp/big")
z.write("/tmp/big")
z.close()
#------------------------------------------------------------------------


Regards,
Bengt Richter
Jul 18 '05 #5
RM
>From what I understand, Winzip uses their own proprietary version of a
Zip format. That means that you will only be able open those archives
with Winzip. (IOW, you are locked in)

I think someone mentioned having developed a Python module for the 7zip
fomat, but I may be wrong. In any case, you could simply write a
wrapper over the command line version of 7zip and your problem would be
solved. There is a Linux version as well.
(http://p7zip.sourceforge.net/)
bennie wrote:
Christos TZOTZIOY Georgiou wrote:
AFAIR there is a 4GiB (or 2GiB) size limit applying both to files added to zip and to the total size of the zip file. This limit comes from the zip file specification (32 bit offsets).
Can it be that you are creating a zip file that its total size exceeds the limit?

That is possible.
But with Winzip program it can.
How come that is with ZipFile not works


Jul 18 '05 #6
Bennie wrote:
How can I fix this?


Since this is the question that you actually asked, there
is an easy answer. Investigate the problem in detail, understand
the source code of the zipfile module, and the format of zip files,
and develop a change to the code to correct the behaviour.
Then submit a patch to sf.net/projects/python.

Regards,
Martin
Jul 18 '05 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
1966
by: Dennis Hotson | last post by:
I'm having trouble adding a file to a .zip file using python2.3. The write method of a ZipFile object needs a filename in order to add a file to the archive. The problem is that I want to add a 'file-like' object... ie the 'file' doesn't have a real filename. Any suggestions on what I could do? :) Oh and also, I looked at os.tmpname()...
0
1382
by: Ian Cook | last post by:
Hi, I'm new to python so please excuse me if this is a 'dumb' question. I want to use the ZipFile module to UPDATE, rather than append, or create a new zip file. Does anyone know how this can be done? This is a snippet of the code I have now..
8
1924
by: perchef | last post by:
hi, i have written this small code which allow me to unzip files using python : import zipfile import os import os.path pathDir="/home/toto/Desktop"
7
3347
by: Chris | last post by:
Hi Where can I find info on unzipping file with VB.NET. I need to unzip a winzip file with my application Thanks
12
5709
by: pac | last post by:
I'm preparing to distribute a Windows XP Python program and some ancillary files, and I wanted to put everything in a .ZIP archive. It proved to be inordinately difficult and I thought I would post my solution here. Is there a better one? Suppose you have a set of files in a directory c:\a\b and some additional files in c:\a\b\subdir....
1
6394
by: Ritesh Raj Sarraf | last post by:
Hi, The program downloads the files from the internet and compresses them to a single zip archive using compress_the_file(). Upon running syncer() which calls the decompress_the_file(), the first iteration succeeds. But upon second iteration, I get an IOError exception with the message: "compressed file ended before the logical...
7
7579
by: erikcw | last post by:
Hi all, I'm trying to extract zip file (containing an xml file) from an email so I can process it. But I'm running up against some brick walls. I've been googling and reading all afternoon, and can't seem to figure it out. Here is what I have so far. p = POP3("mail.server.com")
9
7369
by: flebber | last post by:
I was working at creating a simple program that would read the content of a playlist file( in this case *.k3b") and write it out . the compressed "*.k3b" file has two file and the one I was trying to read was maindata.xml . I cannot however seem to use the gzip module correctly. Have tried the program 2 ways for no success, any ideas would be...
5
2215
by: techusky | last post by:
I made a script that successfully creates a .zip file of all the files in a directory on my web server, but now what I haven't figured out how to do is how to have it automatically deleted when the user successfully downloads it, as otherwise my server would eventually get clogged up with all these zip files. Any help/suggestions? Thanks
0
7355
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
1
7356
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
7697
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5285
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
4900
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3395
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3395
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
1830
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
1
979
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.