module import performance question

[Jeff Sykes]

I have a cgi script that imports modules based on a user config file.
I am suffering some performance problems when I import these modules.
Some logging revealed that it seems to be taking about 1.3 seconds to
import these modules. This is running on a Windows 2000 box with a
Pentium II 400 processor, Python 2.1, Apache 1.3.19 as the web server.
I don't need screaming performance numbers, but this time is
excessive. Is it me, or does that number seem kind of slow? Any code
optimizations or recommendations? I would like to keep the
architecture the same, just standard cgi scripts, so mod_python looks
like it's a bit more involved than I would like.

Here is a code snip:

dyn_blocks = {} # set up a dict of the dynamic imports
if confok: # was the configuration present?
blocks = getBlocks(config) # get the application info from config
for block in blocks: # each block indictates the ap & pg (module)
ap = string.split(string.strip(block[1:len(block)]))[0]
pg = string.split(string.strip(block[1:len(block)]))[1]
sys.path.append(getAppPath(ap)) # add module dir to sys.path
if (block[0] == '~'):
try:
dyn_blocks[ap + "_" + pg] = __import__(pg)
except:
Log.stacktrace(Log.ERROR) # log problem importing module

Any help is appreciated.

[Emile van Sebille]

"Jeff Sykes" <je**@cowz.com> wrote in message
news:72**************************@posting.google.c om...

I have a cgi script that imports modules based on a user config file. I am suffering some performance problems when I import these modules. Some logging revealed that it seems to be taking about 1.3 seconds to import these modules. This is running on a Windows 2000 box with a
Pentium II 400 processor, Python 2.1, Apache 1.3.19 as the web server. I don't need screaming performance numbers, but this time is
excessive. Is it me, or does that number seem kind of slow?
How many modules are being imported?
Any code
optimizations or recommendations?

Don't append to sys.path. Try sys.path.insert(0,addlpath). Then the
import will get an immediate hit and always the right module rather
than stepping through all the preceeding directories.

HTH,
Emile van Sebille
em***@fenx.com

[Jeff Sykes]

Thanks, Emile, that's a good idea. I found another issue, too. See my
response to another poster below:

I think there was a different culprit, though. After some more
dilligent logging, I realized that immediately after import I
called upon an attribute of the imported module that wouldn't
_necessarily_ be present. When the attribute was not present,
it raised an exception, which I caught. This exception handling
was a real performance anchor! I now check hasattr() before I
execute this block of code, and this has helped performance
significantly.

Here's a snip of the block that occured right after import:

try:
# next line was not there before
if hasattr(dyn_blocks[key], "BUFFER_ON"):
context.setBufferFlag(dyn_blocks[key].BUFFER_ON)
except:
Log.stacktrace(Log.ERROR)

I guess the lesson I learned is don't use exception handling
unnecessarily. Which I know not to do anyway. Sigh. Back to
school for me.

"Emile van Sebille" <em***@fenx.com> wrote in message news:<bo*************@ID-11957.news.uni-berlin.de>... "Jeff Sykes" <je**@cowz.com> wrote in message
news:72**************************@posting.google.c om...

I have a cgi script that imports modules based on a user config

file.

I am suffering some performance problems when I import these

modules.

Some logging revealed that it seems to be taking about 1.3 seconds

to

import these modules. This is running on a Windows 2000 box with a
Pentium II 400 processor, Python 2.1, Apache 1.3.19 as the web

server.

I don't need screaming performance numbers, but this time is
excessive. Is it me, or does that number seem kind of slow?

How many modules are being imported?

Any code
optimizations or recommendations?

Don't append to sys.path. Try sys.path.insert(0,addlpath). Then the
import will get an immediate hit and always the right module rather
than stepping through all the preceeding directories.

HTH,
Emile van Sebille
em***@fenx.com

[Peter Abel]

je**@cowz.com (Jeff Sykes) wrote in message news:<72**************************@posting.google. com>...
....
....
....

Here is a code snip:

dyn_blocks = {} # set up a dict of the dynamic imports
if confok: # was the configuration present?
blocks = getBlocks(config) # get the application info from config
for block in blocks: # each block indictates the ap & pg (module)
doen't know how long is len(blocks), if it's very long you could
accelerate the *for loop* be replacing the following two lines with:

(ap,pg) = string.split(string.strip(block[1:]))
**Remark: block[1:len(block)] is the same as block[1:], but without
calulating len(block) everytime, further more the split
and the strip is done only once
ap = string.split(string.strip(block[1:len(block)]))[0]
pg = string.split(string.strip(block[1:len(block)]))[1]

....
....
....

Regards
Peter

[Scott D. Daniels]

je**@cowz.com (Jeff Sykes) wrote in message news:<72**************************@posting.google. com>...

...
Here's a snip of the block that occured right after import:

try:
# next line was not there before
if hasattr(dyn_blocks[key], "BUFFER_ON"):
context.setBufferFlag(dyn_blocks[key].BUFFER_ON)
except:
Log.stacktrace(Log.ERROR)
I guess the lesson I learned is don't use exception handling
unnecessarily. Which I know not to do anyway. Sigh. Back to
school for me.
Wrong lesson. Python wants you to use exception handling.
The time sink above code is the unnecessary uses of: Log.stacktrace(Log.ERROR)

A couple of Alternatives:
## smaller, not quite careful enough for my taste
try:
context.setBufferFlag(dyn_blocks[key].BUFFER_ON)
except AttributeError:
pass
except:
Log.stacktrace(Log.ERROR)

## better, more to my taste
try:
try:
flag = dyn_blocks[key].BUFFER_ON
except AttributeError:
pass
else:
context.setBufferFlag(flag)
except:
Log.stacktrace(Log.ERROR)

-Scott