473,390 Members | 867 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,390 software developers and data experts.

Where can I suggest an enchantment for Python Zip lib?

Hi!

Where can I ask it?

I want to ask that developers change the Python's Zip lib in the next
versions.
The Zip lib not have a callback procedure. When I zip something, I don't
know, what is the actual position of the processing, and how many bytes
remaining.
It is simply rewriteable, but when I get new Python, it is forget this
thing again...

So some callback needed for it, if possible. With this I can abort
processing and I can show the actual state when I processing a large file.

See this thread:
http://groups.google.com.kh/group/co...b9b1c286d9af7b

Thanks for your help:
dd
Jun 7 '07 #1
9 1401
durumdara wrote:
Hi!

Where can I ask it?

I want to ask that developers change the Python's Zip lib in the next
versions.
The Zip lib not have a callback procedure. When I zip something, I don't
know, what is the actual position of the processing, and how many bytes
remaining.
It is simply rewriteable, but when I get new Python, it is forget this
thing again...

So some callback needed for it, if possible. With this I can abort
processing and I can show the actual state when I processing a large file.

See this thread:
http://groups.google.com.kh/group/co...b9b1c286d9af7b
Thanks for your help:
dd

You can easily find out roughly how many bytes are in your .ZIP archive
by using following:

zipbytes=Zobj.fp.tell()

Where Zobj is your zipfile instance. You don't need a callback.

Problem is ill defined for a better solution. You don't know how much
the "next" file will compress. It may compress a lot, not at all or
in some situations actually grow. So it is difficult (impossible?) to
know how many bytes are remaining. I have a rough calculation where
I limit the files to 2Gb, but you must set aside some space for the
table of contents that gets added at the end (whose size you don't
actually know either). So I use:

maxzipbytesupperlimit=int((1L<<31)-(8*(1<<20)))

That is 2Gb-8Mb maximum TOC limit of a zip file.

I look at zipbytes add the uncompressed size of the next file, if it
exceeds maxzipbytesupperlimit, I close the file and move to the next
zip archive. If it is smaller, I add the file to the archive.

Hope this helps.

-Larry
Jun 7 '07 #2
durumdara wrote:
Hi!

Where can I ask it?

I want to ask that developers change the Python's Zip lib in the next
versions.
The Zip lib not have a callback procedure. When I zip something, I don't
know, what is the actual position of the processing, and how many bytes
remaining.
It is simply rewriteable, but when I get new Python, it is forget this
thing again...

So some callback needed for it, if possible. With this I can abort
processing and I can show the actual state when I processing a large file.

See this thread:
http://groups.google.com.kh/group/co...b9b1c286d9af7b
Thanks for your help:
dd

You can easily find out roughly how many bytes are in your .ZIP archive
by using following:

zipbytes=Zobj.fp.tell()

Where Zobj is your zipfile instance. You don't need a callback.

Problem is ill defined for a better solution. You don't know how much
the "next" file will compress. It may compress a lot, not at all or
in some situations actually grow. So it is difficult (impossible?) to
know how many bytes are remaining. I have a rough calculation where
I limit the files to 2Gb, but you must set aside some space for the
table of contents that gets added at the end (whose size you don't
actually know either). So I use:

maxzipbytesupperlimit=int((1L<<31)-(8*(1<<20)))

That is 2Gb-8Mb maximum TOC limit of a zip file.

I look at zipbytes add the uncompressed size of the next file, if it
exceeds maxzipbytesupperlimit, I close the file and move to the next
zip archive. If it is smaller, I add the file to the archive.

Hope this helps.

-Larry

Jun 7 '07 #3
Hi Larry!
durumdara wrote:
You can easily find out roughly how many bytes are in your .ZIP archive
by using following:

zipbytes=Zobj.fp.tell()
The main problem is not this.
I want to write a backup software, and I want to:
- see the progress in the processing of the actual file
- abort the progress if I need it
If I compress small files, I don't have problems.
But with larger files (10-20 MB) I get problems, because the zipfile's
method is uninterruptable.
Only one way I have to control this: if I modify the ZipFile module.

dd

On Jun 7, 8:26 pm, Larry Bates <larry.ba...@websafe.comwrote:
Where Zobj is your zipfile instance. You don't need a callback.

Problem is ill defined for a better solution. You don't know how much
the "next" file will compress. It may compress a lot, not at all or
in some situations actually grow. So it is difficult (impossible?) to
know how many bytes are remaining. I have a rough calculation where
I limit the files to 2Gb, but you must set aside some space for the
table of contents that gets added at the end (whose size you don't
actually know either). So I use:

maxzipbytesupperlimit=int((1L<<31)-(8*(1<<20)))

That is 2Gb-8Mb maximum TOC limit of a zip file.

I look at zipbytes add the uncompressed size of the next file, if it
exceeds maxzipbytesupperlimit, I close the file and move to the next
zip archive. If it is smaller, I add the file to the archive.

Hope this helps.

-Larry

Jun 8 '07 #4
In article <ma***************************************@python. org>,
durumdara <du*******@gmail.comwrote:
>
[...]
Click your heels together three times and say, "Abracadabra!"
(Sorry, couldn't resist.)
--
Aahz (aa**@pythoncraft.com) <* http://www.pythoncraft.com/

"as long as we like the same operating system, things are cool." --piranha
Jun 9 '07 #5
Hogwarts.

Sorry, I couldn't resist either.

I'm sure you meant to say "enhancement" - an "enchantment" is a magic
spell, often used to lull an unsuspecting victim into some sort of
compliance or trance. Actually, if you have an *enchantment* for
Python, I'm sure several people on this list would be interested. :)

But yes, *enhancement* requests can be posted on the SF feature
request list. You may need to more clearly define what kind of data
these zlib callbacks would receive, and under what conditions they
would be called.

-- Paul

Jun 9 '07 #6
durumdara wrote:
Hi Larry!
>durumdara wrote:
You can easily find out roughly how many bytes are in your .ZIP archive
by using following:

zipbytes=Zobj.fp.tell()

The main problem is not this.
I want to write a backup software, and I want to:
- see the progress in the processing of the actual file
- abort the progress if I need it
If I compress small files, I don't have problems.
But with larger files (10-20 MB) I get problems, because the zipfile's
method is uninterruptable.
Only one way I have to control this: if I modify the ZipFile module.

dd

On Jun 7, 8:26 pm, Larry Bates <larry.ba...@websafe.comwrote:
On my 3 year old 3Ghz Pentium III it takes about 8 seconds to zip 20Mb file.
So what is the problem? Not updating the process for 8-10 seconds
should be just fine for most applications.

-Larry
Jun 9 '07 #7
durumdara wrote:
Only one way I have to control this: if I modify the ZipFile module.
Since you already have the desired feature implemented, why don't you submit a
patch.

See http://www.python.org/patches/

- Anders
Jun 10 '07 #8
On my 3 year old 3Ghz Pentium III it takes about 8 seconds to zip 20Mb file.
So what is the problem? Not updating the process for 8-10 seconds
should be just fine for most applications.

-Larry
The problem, that:
- I want to process 100-200 MB zip files.
- I want to abort in process
- I want to know the actual position
- I want to slow the operation sometimes!

Why I want to slow?

The big archiving is slow operation. When it is slow, I want to
working with other apps while it is processing.

So I want to slow the zipping with time.sleep, then the background
thread is not use the full CPU...
I can work with other apps.

dd

Jun 13 '07 #9
durumdara <du*******@gmail.comwrote:
>
The problem, that:
- I want to process 100-200 MB zip files.
- I want to abort in process
- I want to know the actual position
- I want to slow the operation sometimes!

Why I want to slow?

The big archiving is slow operation. When it is slow, I want to
working with other apps while it is processing.

So I want to slow the zipping with time.sleep, then the background
thread is not use the full CPU...
I can work with other apps.
Surely a more intelligent (and much easier) solution would be to reduce the
priority of your archiving process. Your operating system is far better
equipped to share the CPU than you are.
--
Tim Roberts, ti**@probo.com
Providenza & Boekelheide, Inc.
Jun 15 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Helmut Jarausch | last post by:
Hi, when I say e.g. help('try') in an interactive Python (or idle) session I get Sorry, topic and keyword documentation is not available because the Python HTML documentation files could not...
9
by: santanu | last post by:
Hi all, I know a little python (not the OOP part) learnt by studying the online tutorial. Now I would like to learn it more thoroughly. I have access to 'Programming Python' which I liked...
25
by: Tor Erik Sønvisen | last post by:
Hi I need to browse the socket-module source-code. I believe it's contained in the file socketmodule.c, but I can't locate this file... Where should I look? regards tores
1
by: mirandacascade | last post by:
Version of python: 2.4 O/S: Win2K I will be writing some python scripts to do some client-side programming that involves socket.py. I expect that I will be making calls to the following...
36
by: Alex Martelli | last post by:
So, I thought I'd tool up to let me build and test Python extensions on Windows (as well as Mac and Linux) -- I'm trying out Parallels Workstation beta on my new Macbook Pro (and so far it seems to...
23
by: gord | last post by:
As a complete novice in the study of Python, I am asking myself where this language is superior or better suited than others. For example, all I see in the tutorials are lots of examples of list...
5
by: Wijaya Edward | last post by:
I tried to call pydoc from my Linux box. To my realization that it doesn't contain pydoc. I thought this module should be a default module. Can anybody suggest where can I find the module? I...
33
by: NicolasG | last post by:
Hi, I want to be a professional python programmer, unfortunately I'm working on technical support and don't have the time/patience to start making projects my self. I tried to apply to some...
8
by: jmDesktop | last post by:
I have been to the main python site, but am still confused. I have been using .net, so it may be obvious how to do this to everyone else. I am aware there are various frameworks (Django, Pylons,...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: ryjfgjl | last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
0
by: ryjfgjl | last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.