Speed ain't bad

[Bulba!]

One of the posters inspired me to do profiling on my newbie script
(pasted below). After measurements I have found that the speed
of Python, at least in the area where my script works, is surprisingly
high.

This is the experiment: a script recreates the folder hierarchy
somewhere else and stores there the compressed versions of
files from source hierarchy (the script is doing additional backups
of the disk of file server at the company where I work onto other
disks, with compression for sake of saving space). The data was:

468 MB, 15057 files, 1568 folders
(machine: win2k, python v2.3.3)

The time that WinRAR v3.20 (with ZIP format and normal compression
set) needed to compress all that was 119 seconds.

The Python script time (running under profiler) was, drumroll...

198 seconds.

Note that the Python script had to laboriously recreate the tree of
1568 folders and create over 15 thousand compressed files, so
it had more work to do actually than WinRAR did. The size of
compressed data was basically the same, about 207 MB.

I find it very encouraging that in the real world area of application
a newbie script written in the very high-level language can have the
performance that is not that far from the performance of "shrinkwrap "
pro archiver (WinRAR is excellent archiver, both when it comes to
compression as well as speed). I do realize that this is mainly
the result of all the "underlying infrastructure" of Python. Great
work, guys. Congrats.

The only thing I'm missing in this picture is knowledge if my script
could be further optimised (not that I actually need better
performance, I'm just curious what possible solutions could be).

Any takers among the experienced guys?

Profiling results:

p3.sort_stats(' cumulative').pr int_stats(40)

Fri Dec 31 01:04:14 2004 p3.tmp

580543 function calls (568607 primitive calls) in 198.124 CPU
seconds

Ordered by: cumulative time
List reduced from 69 to 40 due to restriction <40>

ncalls tottime percall cumtime percall
filename:lineno (function)
1 0.013 0.013 198.124 198.124 profile:0(z3())
1 0.000 0.000 198.110 198.110 <string>:1(?)
1 0.000 0.000 198.110 198.110 <interactive
input>:1(z3)
1 1.513 1.513 198.110 198.110 zmtree3.py:26(z mtree)
15057 14.504 0.001 186.961 0.012 zmtree3.py:7(zf )
15057 147.582 0.010 148.778 0.010
C:\Python23\lib \zipfile.py:388 (write)
15057 12.156 0.001 12.156 0.001
C:\Python23\lib \zipfile.py:182 (__init__)
32002 7.957 0.000 8.542 0.000
C:\PYTHON23\Lib \ntpath.py:266( isdir)
13826/1890 2.550 0.000 8.143 0.004
C:\Python23\lib \os.py:206(walk )
30114 3.164 0.000 3.164 0.000
C:\Python23\lib \zipfile.py:483 (close)
60228 1.753 0.000 2.149 0.000
C:\PYTHON23\Lib \ntpath.py:157( split)
45171 0.538 0.000 2.116 0.000
C:\PYTHON23\Lib \ntpath.py:197( basename)
15057 1.285 0.000 1.917 0.000
C:\PYTHON23\Lib \ntpath.py:467( abspath)
33890 0.688 0.000 1.419 0.000
C:\PYTHON23\Lib \ntpath.py:58(j oin)
109175 0.783 0.000 0.783 0.000
C:\PYTHON23\Lib \ntpath.py:115( splitdrive)
15057 0.196 0.000 0.768 0.000
C:\PYTHON23\Lib \ntpath.py:204( dirname)
33890 0.433 0.000 0.731 0.000
C:\PYTHON23\Lib \ntpath.py:50(i sabs)
15057 0.544 0.000 0.632 0.000
C:\PYTHON23\Lib \ntpath.py:438( normpath)
32002 0.431 0.000 0.585 0.000
C:\PYTHON23\Lib \stat.py:45(S_I SDIR)
15057 0.555 0.000 0.555 0.000
C:\Python23\lib \zipfile.py:149 (FileHeader)
15057 0.483 0.000 0.483 0.000
C:\Python23\lib \zipfile.py:116 (__init__)
151 0.002 0.000 0.435 0.003
C:\PYTHON23\lib \site-packages\Python win\pywin\frame work\winout.py: 171(write)
151 0.002 0.000 0.432 0.003
C:\PYTHON23\lib \site-packages\Python win\pywin\frame work\winout.py: 489(write)
151 0.013 0.000 0.430 0.003
C:\PYTHON23\lib \site-packages\Python win\pywin\frame work\winout.py: 461(HandleOutpu t)
76 0.087 0.001 0.405 0.005
C:\PYTHON23\lib \site-packages\Python win\pywin\frame work\winout.py: 430(QueueFlush)
15057 0.239 0.000 0.340 0.000
C:\Python23\lib \zipfile.py:479 (__del__)
15057 0.157 0.000 0.157 0.000
C:\Python23\lib \zipfile.py:371 (_writecheck)
32002 0.154 0.000 0.154 0.000
C:\PYTHON23\Lib \stat.py:29(S_I FMT)
76 0.007 0.000 0.146 0.002
C:\PYTHON23\lib \site-packages\Python win\pywin\frame work\winout.py: 262(dowrite)
76 0.007 0.000 0.137 0.002
C:\PYTHON23\lib \site-packages\Python win\pywin\scint illa\formatter. py:221(OnStyleN eeded)
76 0.011 0.000 0.118 0.002
C:\PYTHON23\lib \site-packages\Python win\pywin\frame work\interact.p y:197(Colorize)
76 0.110 0.001 0.112 0.001
C:\PYTHON23\lib \site-packages\Python win\pywin\scint illa\control.py :69(SCIInsertTe xt)
76 0.079 0.001 0.081 0.001
C:\PYTHON23\lib \site-packages\Python win\pywin\scint illa\control.py :333(GetTextRan ge)
76 0.018 0.000 0.020 0.000
C:\PYTHON23\lib \site-packages\Python win\pywin\scint illa\control.py :296(SetSel)
76 0.006 0.000 0.018 0.000
C:\PYTHON23\lib \site-packages\Python win\pywin\scint illa\document.p y:149(__call__)
227 0.003 0.000 0.012 0.000
C:\Python23\lib \Queue.py:172(g et_nowait)
76 0.007 0.000 0.011 0.000
C:\PYTHON23\lib \site-packages\Python win\pywin\frame work\interact.p y:114(ColorizeI nteractiveCode)
532 0.011 0.000 0.011 0.000
C:\PYTHON23\lib \site-packages\Python win\pywin\scint illa\control.py :330(GetTextLen gth)
76 0.001 0.000 0.010 0.000
C:\PYTHON23\lib \site-packages\Python win\pywin\scint illa\view.py:25 6(OnBraceMatch)
1888 0.009 0.000 0.009 0.000
C:\PYTHON23\Lib \ntpath.py:245( islink)
---
Script:

#!/usr/bin/python

import os
import sys
from zipfile import ZipFile, ZIP_DEFLATED

def zf(sfpath, targetdir):
if (sys.platform[:3] == 'win'):
tgfpath=sfpath[2:]
else:
tgfpath=sfpath
zfdir=os.path.d irname(os.path. abspath(targetd ir) + tgfpath)
zfpath=zfdir + os.path.sep + os.path.basenam e(tgfpath) + '.zip'
if(not os.path.isdir(z fdir)):
os.makedirs(zfd ir)
archive=ZipFile (zfpath, 'w', ZIP_DEFLATED)
sfile=open(sfpa th,'rb')
zfname=os.path. basename(tgfpat h)
archive.write(s fpath, os.path.basenam e(zfpath), ZIP_DEFLATED)
archive.close()
ssize=os.stat(s fpath).st_size
zsize=os.stat(z fpath).st_size
return (ssize,zsize)
def zmtree(sdir,tdi r):
n=0
ssize=0
zsize=0
sys.stdout.writ e('\n ')
for root, dirs, files in os.walk(sdir):
for file in files:
res=zf(os.path. join(root,file) ,tdir)
ssize+=res[0]
zsize+=res[1]
n=n+1
#sys.stdout.wri te('.')
if (n % 200 == 0):
print " %.2fM (%.2fM)" % (ssize/1048576.0,
zsize/1048576.0)
#sys.stdout.wri te(' ')
return (n, ssize, zsize)
if __name__=="__ma in__":
if len(sys.argv) == 3:
if(os.path.isdi r(sys.argv[1]) and os.path.isdir(s ys.argv[2])):

(n,ssize,zsize) =zmtree(os.path .abspath(sys.ar gv[1]),os.path.abspa th(sys.argv[2]))
print "\n\n Summary:\n Number of files compressed: %d\n
Total size of original files: %.2fM\n \
Total size of compressed files: %.2fM" % (n, ssize/1048576.0,
zsize/1048576.0)
sys.exit(0)
else:
print "Incorrect arguments."
if (not os.path.isdir(s ys.argv[1])): print sys.argv[1] + "
is not directory."
if (not os.path.isdir(s ys.argv[2])): print sys.argv[2] + "
is not directory."

print "\n Usage:\n " + sys.argv[0] + " source-directory
target-directory"

--
It's a man's life in a Python Programming Association.

[Anders J. Munch]

"Bulba!" <bu***@bulba.co m> wrote:

One of the posters inspired me to do profiling on my newbie script
(pasted below). After measurements I have found that the speed
of Python, at least in the area where my script works, is surprisingly
high.
Pretty good code for someone who calls himself a newbie.

One line that puzzles me: sfile=open(sfpa th,'rb')
You never use sfile again.
In any case, you should explicitly close all files that you open. Even
if there's an exception:

sfile = open(sfpath, 'rb')
try:
<stuff to do with the file open>
finally:
sfile.close()

The only thing I'm missing in this picture is knowledge if my script
could be further optimised (not that I actually need better
performance, I'm just curious what possible solutions could be).

Any takers among the experienced guys?

Basically the way to optimise these things is to cut down on anything
that does I/O: Use as few calls to os.path.is{dir, file}, os.stat, open
and such that you can get away with.

One way to do that is caching; e.g. storing names of known directories
in a set (sets.Set()) and checking that set before calling
os.path.isdir. I haven't spotted any obvious opportunities for that
in your script, though.

Another way is the strategy of "it's easier to ask forgiveness than to
ask permission".
If you replace:
if(not os.path.isdir(z fdir)):
os.makedirs(zfd ir)
with:
try:
os.makedirs(zfd ir)
except EnvironmentErro r:
pass

then not only will your script become a micron more robust, but
assuming zfdir typically does not exist, you will have saved the call
to os.path.isdir.

- Anders

[Bulba!]

On Sat, 1 Jan 2005 14:20:06 +0100, "Anders J. Munch"
<an******@inbou nd.dk> wrote:

One of the posters inspired me to do profiling on my newbie script
(pasted below). After measurements I have found that the speed
of Python, at least in the area where my script works, is surprisingly
high.
Pretty good code for someone who calls himself a newbie.

<blush>
One line that puzzles me:

sfile=open(sfpa th,'rb')

You never use sfile again.
Right! It's a leftover from a previous implementation (that
used bzip2). Forgot to delete it, thanks.
Another way is the strategy of "it's easier to ask forgiveness than to
ask permission".
If you replace:
if(not os.path.isdir(z fdir)):
os.makedirs(zfd ir)
with:
try:
os.makedirs(zfd ir)
except EnvironmentErro r:
pass then not only will your script become a micron more robust, but
assuming zfdir typically does not exist, you will have saved the call
to os.path.isdir.

Yes, this is the kind of habit that low-level languages like C
missing features like exceptions ingrain in a mind of a programmer...

Getting out of this straitjacket is kind of hard - it would not cross
my mind to try smth like what you showed me, thanks!

Exceptions in Python are a GODSEND. I strongly recommend
to any former C programmer wanting to get rid of a "straightjacket "
to read the following to get an idea how not to write C code in Python
and instead exploit the better side of VHLL:

http://gnosis.cx/TPiP/appendix_a.txt


--
It's a man's life in a Python Programming Association.

[Jeff Shannon]

Anders J. Munch wrote:

Another way is the strategy of "it's easier to ask forgiveness than to
ask permission".
If you replace:
if(not os.path.isdir(z fdir)):
os.makedirs(zfd ir)
with:
try:
os.makedirs(zfd ir)
except EnvironmentErro r:
pass

then not only will your script become a micron more robust, but
assuming zfdir typically does not exist, you will have saved the call
to os.path.isdir.

.... at the cost of an exception frame setup and an incomplete call to
os.makedirs(). It's an open question whether the exception setup and
recovery take less time than the call to isdir(), though I'd expect
probably not. The exception route definitely makes more sense if the
makedirs() call is likely to succeed; if it's likely to fail, then
things are murkier.

Since isdir() *is* a disk i/o operation, then in this case the
exception route is probably preferable anyhow. In either case, one
must touch the disk; in the exception case, there will only ever be
one disk access (which either succeeds or fails), while in the other
case, there may be two disk accesses. However, if it wasn't for the
extra disk i/o operation, then the 'if ...' might be slightly faster,
even though the exception-based route is more Pythonic.

Jeff Shannon
Technician/Programmer
Credit International

[John Machin]

Anders J. Munch wrote:

Another way is the strategy of "it's easier to ask forgiveness than to ask permission".
If you replace:
if(not os.path.isdir(z fdir)):
os.makedirs(zfd ir)
with:
try:
os.makedirs(zfd ir)
except EnvironmentErro r:
pass

then not only will your script become a micron more robust, but
assuming zfdir typically does not exist, you will have saved the call
to os.path.isdir.

1. Robustness: Both versions will "crash" (in the sense of an unhandled
exception) in the situation where zfdir exists but is not a directory.
The revised version just crashes later than the OP's version :-(
Trapping EnvironmentErro r seems not very useful -- the result will not
distinguish (on Windows 2000 at least) between the 'existing dir' and
'existing non-directory' cases.
Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] on
win32

import os, os.path
os.path.exists( 'fubar_not_dir' ) True os.path.isdir(' fubar_not_dir') False os.makedirs('fu bar_not_dir') Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "c:\Python24\li b\os.py", line 159, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: 'fubar_not_dir' try: .... os.mkdir('fubar _not_dir')
.... except EnvironmentErro r:
.... print 'trapped env err'
....
trapped env err os.mkdir('fubar _is_dir')
os.mkdir('fubar _is_dir') Traceback (most recent call last):
File "<stdin>", line 1, in ?
OSError: [Errno 17] File exists: 'fubar_is_dir'

2. Efficiency: I don't see the disk I/O inefficiency in calling
os.path.isdir() before os.makedirs() -- if the relevant part of the
filesystem wasn't already in memory, the isdir() call would make it so,
and makedirs() would get a free ride, yes/no?

[Anders J. Munch]

"John Machin" <sj******@lexic on.net> wrote:

1. Robustness: Both versions will "crash" (in the sense of an unhandled
2. Efficiency: I don't see the disk I/O inefficiency in calling
3. Don't itemise perceived flaws in other people's postings. It may
give off a hostile impression.
1. Robustness: Both versions will "crash" (in the sense of an unhandled
exception) in the situation where zfdir exists but is not a directory.
The revised version just crashes later than the OP's version :-(
Trapping EnvironmentErro r seems not very useful -- the result will not
distinguish (on Windows 2000 at least) between the 'existing dir' and
'existing non-directory' cases.
Good point; my version has room for improvement. But at least it fixes
the race condition between isdir and makedirs.

What I like about EnvironmentErro r is that it it's easier to use than
figuring out which one of IOError or OSError applies (and whether that
can be relied on, cross-platform).
2. Efficiency: I don't see the disk I/O inefficiency in calling
os.path.isdir() before os.makedirs() -- if the relevant part of the
filesystem wasn't already in memory, the isdir() call would make it
so, and makedirs() would get a free ride, yes/no?

Perhaps. Looking stuff up in operating system tables and buffers takes
time too. And then there's network latency; how much local caching do
you get for an NFS mount or SMB share?

If you really want to know, measure.

- Anders