473,392 Members | 1,347 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,392 software developers and data experts.

Zipfile content reading via an iterator?

I'm dealing with several large items that have been zipped up to
get quite impressive compression. However, uncompressed, they're
large enough to thrash my memory to swap and in general do bad
performance-related things. I'm trying to figure out how to
produce a file-like iterator out of the contents of such an item.
>>z = zipfile.zipFile("test.zip")
info = z.getinfo("data.txt")
info.compress_size
132987864
>>info.file_size
1344250972
>>len(z.namelist())
20

I need to be able to access multiple files within it, but I can
iterate over each one, only seeing small slices of the file.
Using the read() method triggers the volumnous read. Thus what I
have to do currently:
>>content = z.read("data.txt") # ouch!
len(content)
1344250972
>>for row in content.splitlines(): process(row) # pain!
What I'm trying to figure out how to do is something like the
mythical:
>>for row in z.file_iter("data.txt"): process(row) # aah
to more efficiently handle the huge stream of data.

Am I missing something obvious? It seems like iterating over zip
contents would be a common thing to do (especially when compared
to reading the whole contents...I mean, they're zipped because
they're big! :)

Thanks for any pointers,

-tkc

Dec 11 '07 #1
0 1005

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

6
by: Bennie | last post by:
Hi, I have a problem with ZipFile. It works okay untily I come across a file that is greater then 1Gb. Then it exit with the error: OverflowError: long int too large to convert to int How...
2
by: Renzo | last post by:
Hi, I'm working to create a backup function for a software; in particular, from a directory with 12300 files (Jpg, medium size = 250KB / total size = 2.90GB), i have to create a zip file. I've...
3
by: rzed | last post by:
I create a zip file on my WinXP system, using this function: <fn> import zipfile import os import os.path def zipdir(dirname, zfname): zf = zipfile.ZipFile(zfname, 'w',...
11
by: Hari Sekhon | last post by:
I do import zipfile zip=zipfile.ZipFile('d:\somepath\cdimage.zip') zip.namelist() then either of the two: A) file('someimage.iso','w').write(zip.read('someimage.iso'))
8
by: =?utf-8?B?5Lq66KiA6JC95pel5piv5aSp5rav77yM5pyb5p6B | last post by:
I made a C/S network program, the client receive the zip file from the server, and read the data into a variable. how could I process the zipfile directly without saving it into file. In the...
5
by: Martin | last post by:
I get below error when trying to write unicode xml to a zipfile. zip.writestr('content.xml', content.toxml()) File "/usr/lib/python2.4/zipfile.py", line 460, in writestr zinfo.CRC =...
12
by: xamdam | last post by:
Hi fellas, I am experiencing problems reading a 2GB zipfile consisting of multiple zipped files. I found a thread http://mail.python.org/pipermail/python-dev/2005-April/053027.html that mentions...
5
by: Neil Crighton | last post by:
I'm using the zipfile library to read a zip file in Windows, and it seems to be adding too many newlines to extracted files. I've found that for extracted text-encoded files, removing all instances...
2
by: Alan G Isaac | last post by:
I have a large ASCII data set that is zipped to a reasonable size. Can I access the data without decompressing the whole file first? I would like to run through the data to produce a much smaller...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.