469,338 Members | 8,589 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,338 developers. It's quick & easy.

Zipfile content reading via an iterator?

I'm dealing with several large items that have been zipped up to
get quite impressive compression. However, uncompressed, they're
large enough to thrash my memory to swap and in general do bad
performance-related things. I'm trying to figure out how to
produce a file-like iterator out of the contents of such an item.
>>z = zipfile.zipFile("test.zip")
info = z.getinfo("data.txt")
info.compress_size
132987864
>>info.file_size
1344250972
>>len(z.namelist())
20

I need to be able to access multiple files within it, but I can
iterate over each one, only seeing small slices of the file.
Using the read() method triggers the volumnous read. Thus what I
have to do currently:
>>content = z.read("data.txt") # ouch!
len(content)
1344250972
>>for row in content.splitlines(): process(row) # pain!
What I'm trying to figure out how to do is something like the
mythical:
>>for row in z.file_iter("data.txt"): process(row) # aah
to more efficiently handle the huge stream of data.

Am I missing something obvious? It seems like iterating over zip
contents would be a common thing to do (especially when compared
to reading the whole contents...I mean, they're zipped because
they're big! :)

Thanks for any pointers,

-tkc

Dec 11 '07 #1
0 854

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

6 posts views Thread by Bennie | last post: by
2 posts views Thread by Renzo | last post: by
3 posts views Thread by rzed | last post: by
11 posts views Thread by Hari Sekhon | last post: by
8 posts views Thread by =?utf-8?B?5Lq66KiA6JC95pel5piv5aSp5rav77yM5pyb5p6B | last post: by
12 posts views Thread by xamdam | last post: by
5 posts views Thread by Neil Crighton | last post: by
2 posts views Thread by Alan G Isaac | last post: by
1 post views Thread by CARIGAR | last post: by
reply views Thread by zhoujie | last post: by
reply views Thread by suresh191 | last post: by
reply views Thread by Marylou17 | last post: by
1 post views Thread by Marylou17 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.