[I tried googling for this, didn't find anything relevant.]
We've recently been doing some profiling on a project of ours. It runs
quite fast on Linux but *really* bogs down on Windows 2003. We initially
thought it was the simplejson libraries (we don't use the C extensions) but
profiling proved otherwise.
We have a function that does some runtime imports via calls to __import__.
We ran 1000 iterations (we used cProfile) of the application (web app).
There were eight calls to __import__ per iteration, so 8000 calls total.
Identical hardware, by the way.
On Linux (Debian Etch, Python 2.5.1)
Total time was 2.793 CPU seconds, with __import__ using 1.059 seconds of
that. So, 37% of the time was spent in import. Not great, but not a show
stopper.
On Windows 2003 (R2, Python 2.5.1)
Total time was 18.532 CPU seconds, with __import__ using 16.330 seconds
(88%) of that.
So, Linux spends 1.734 seconds on non-import activities, and Windows spends
2.202 seconds on non-import activities. Pretty close. But 16.3 seconds on
import!?
Is this a known deficiency in Windows' Python import calls, or is there
something deeper going on here?
Pointers, suggestions, and URLs welcome.
j 4 2113
On Nov 30, 5:08 pm, Joshua Kugler <jkug...@bigfoo t.comwrote:
[I tried googling for this, didn't find anything relevant.]
We've recently been doing some profiling on a project of ours. It runs
quite fast on Linux but *really* bogs down on Windows 2003. We initially
thought it was the simplejson libraries (we don't use the C extensions) but
profiling proved otherwise.
We have a function that does some runtime imports via calls to __import__.
We ran 1000 iterations (we used cProfile) of the application (web app).
There were eight calls to __import__ per iteration, so 8000 calls total.
Identical hardware, by the way.
On Linux (Debian Etch, Python 2.5.1)
Total time was 2.793 CPU seconds, with __import__ using 1.059 seconds of
that. So, 37% of the time was spent in import. Not great, but not a show
stopper.
On Windows 2003 (R2, Python 2.5.1)
Total time was 18.532 CPU seconds, with __import__ using 16.330 seconds
(88%) of that.
So, Linux spends 1.734 seconds on non-import activities, and Windows spends
2.202 seconds on non-import activities. Pretty close. But 16.3 seconds on
import!?
Is this a known deficiency in Windows' Python import calls, or is there
something deeper going on here?
Pointers, suggestions, and URLs welcome.
j
Imagine you have two laundry baskets, one is green, one is purple.
Both contain 10 pairs of pants, but the pockets of the pants in the
purple basket are filled with rocks.
You pick up the green basket - kind of heavy, but not terrible. Then
you pick up the purple basked - wow! Really heavy!
Who would have had any idea that the color of the laundry basket would
make such a difference in the weight? :)
Of course, to clear up this question, you empty both baskets, try to
lift each one, and then find out that they both weigh about the same.
(Or one does weigh more than the other, but now you have ruled out the
possibility that the baskets' contents were a factor in the
comparison.)
Ok, back to your question. __import__ doesn't just load up a module.
At import time, all of the top-level code in the module is executed
also. So if you want to really assess the impact of *just* __import__
on different platforms, then you should try importing:
- empty .py modules
- .py modules containing no top-level executable statements, but some
class or method definitions
- modules that have already been imported
It's possible that your imported module imports additional modules
which have significant platform-dependent logic, or import modules
which are compiled builtins on one platform, but written in Python on
the other.
As it stands, your experiment still has too many unknowns to draw any
decent conclusion, and it certainly is too early to jump to "wow!
__import__ on Windows is sure slower than __import__ on Linux!"
-- Paul
On Dec 1, 10:08 am, Joshua Kugler <jkug...@bigfoo t.comwrote:
[I tried googling for this, didn't find anything relevant.]
We've recently been doing some profiling on a project of ours. It runs
quite fast on Linux but *really* bogs down on Windows 2003. We initially
thought it was the simplejson libraries (we don't use the C extensions) but
profiling proved otherwise.
We have a function that does some runtime imports via calls to __import__.
We ran 1000 iterations (we used cProfile) of the application (web app).
There were eight calls to __import__ per iteration, so 8000 calls total.
Identical hardware, by the way.
On Linux (Debian Etch, Python 2.5.1)
Total time was 2.793 CPU seconds, with __import__ using 1.059 seconds of
that. So, 37% of the time was spent in import. Not great, but not a show
stopper.
On Windows 2003 (R2, Python 2.5.1)
Total time was 18.532 CPU seconds, with __import__ using 16.330 seconds
(88%) of that.
So, Linux spends 1.734 seconds on non-import activities, and Windows spends
2.202 seconds on non-import activities. Pretty close. But 16.3 seconds on
import!?
Is this a known deficiency in Windows' Python import calls, or is there
something deeper going on here?
Pointers, suggestions, and URLs welcome.
j
What modules are you __import__ing, and what is platform-dependent in
each?
What modules are you __import__ing, and what is platform-dependent in
each?
The only thing we're importing __import__ are some modules of ours, with no
sytem dependent code in them at all. Some of them are even empty modules,
as was suggested by one response (for benchmarking purposes).
Turning off Symantec's on-access checking shaved about a bunch of time off
the time spent in __import__ (10 seconds vs 16), but still too high.
Commenting out the import of the only system dependent code we have in our
project (which we don't "manually" import via an __import__ call) produced
no change in run time.
So, we've found a major culprit. I'll see if I can find more.
Thanks for the pointers so far.
j
John Machin wrote:
On Dec 1, 2:12 pm, Joshua Kugler <jkug...@bigfoo t.comwrote:
> x = __import__(m)
Have you ever tried print m, x.__file__ here to check that the modules
are being found where you expect them to be found?
No, I haven't, but I do know for a fact that the only location of the module
found is where I think it is. There are no other modules on the system (or
in the search path) named <ourprefix>_mod ulename.
> except ImportError, e: if not e.message.start swith('No module named'): raise
Why are you blindly ignoring the possibly that the module is not
found? Note that loading a module after it is found is not a zero-cost
operation.
That was in the original code to silently ignore things like a README file
or a directory that didn't have a __init__.py file. The code I posted was
from my benchmarking script. The code in the framework does a listdir()
and goes through that list trying to import. We'll probably "smarten up"
that code a bit, but for now it works.
> x = None
Each of those three module names is a directory under /path/to/code with an empty __init_.py.
Is there anything else in the /path/to/code directory?
A directory and a file that aren't referenced in my test, but are
in the framework's import attempts (see listdir() comment above).
Regardless, setting sys.path to just one directory speeds up both the
benchmark and the framework.
>def r(): for m in ['three','module ','names']: x = __import__(m)
Have you tried print m, x.__file__ here to check that the modules are
being found where you expect them to be found?
As I said above, no I haven't, but I will just to double check.
On Linux, print x, x.__name__ produces the expected result whether I have
one element in sys.path or all of them. On Windows, same result.
Call me crazy, but: First experiment, sys.path was ['/path/to/code',
'', etc etc]. Now it's only ['/path/to/code']. How can that still load
properly but run faster??
I don't know, but it does. Right now, we're just importing empty modules.
Once those modules have imports of their own, my "fix" will probably fall
apart, and we'll be back to square one.
What directory tree walking?? Should be none
if the modules are found in /path/to/code.
I know, that's the crazy thing...there should be no directory tree walking
or other filesystem activity after finding the module. But setting
sys.path to one element does make a difference; that can be seen. I don't
know the internal implementation of __import__, so I can't really comment
beyond my results. Maybe I need to strace a python run.
Are you sure the modules
are always found in /path/to/code? What is in the current directory
[matched by '' in sys.path]?
'' would be whereever I start the benchmark script (~/bin), or whereever the
framework is started from, probably from the web server's home directory,
or from ~/bin when I'm profiling the framework.
Thanks for your responses. I still trying to figure this out too. :)
j This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Marco Herrn |
last post by:
Hi,
I am using the builtin __import__() to import modules. That works for
simple modules like in this example:
m= __import__("eggs")
when there is the module "eggs.py" in the current directory
But how do I do this with packages? A I understand the documentation for
|
by: Coder Coder |
last post by:
Hi,
Can someone help me with how to overload the __import__ function,
so that I can call the old __import__ function and if it cannot find
the library to be able to do something else.
- Thanks.
|
by: bwobbones |
last post by:
Hi,
I'm having trouble making __import__ work with the two classes
attached. The PrintHello() method can't be seen in the BMTest2 class -
what am I doing wrong here?
****************************
class one - BMTest - in BMTest.py:
****************************
import wx
|
by: Steve Juranich |
last post by:
If this is a FAQ, please let me know where the answer is.
I have in some code an 'eval', which I hate, but it's the shortest
path to where I need to get at this point. I thought that one way I
could harden the enviroment against malicious code would be to
temporarily disable the import statement by overloading __import__,
but I tried what seemed obvious to me, and it didn't work.
What I want do do is something like this:
|
by: Thomas Dybdahl Ahle |
last post by:
Hi, I'm writing a gdesklets control, that dynamicly uses __import__ and
getattr to get the right classes based on userinput. The problem is, that
my control is somehow being run from somewhere unknown. This means that
__import__ can't find the modules. I tried putting an
os.chdir("~/.gdesklets/Controls/1136298021.06/...") just before the
__import__ call, and that worked, but ofcource isn't very smart in the run.
Is there any way to find...
| |
by: Mitko Haralanov |
last post by:
Hi all,
I am going to do my best to describe the issue that I am having and
hopefully someone can shed some light on it:
I have three modules that a comprising the problem:
../core.py
../log.py
../resources/simple/__init__.py
|
by: Harold Fellermann |
last post by:
Dear list,
I looked through the list but could not find any solutions for my
current problem.
Within my program, I am importing a module via
__import__(module_name,globals(),locals())
and I want to pass comand line options to this module. I would prefer
not to save them
in a config module or a Config class but rather use the globals and
locals dictionaries
|
by: kaarthikeyapreyan |
last post by:
Query about the __import__ function
~pwd
/home/preyan/test
~ls
sample.py
~python
>>> mod=__import__('sample')
>>> amod=__import__('/home/preyan/test/sample')
>>>
|
by: AmberJain |
last post by:
Windows Autorun FAQs: Description
NOTE- If you are unfamiliar with the concept of autoruns, then read "Windows Autorun FAQs: Overview".
Que-1: How can I safely remove or edit the autorun settings on my PC?
Ans: I recommend that you backup all of your important data before trying anything mentioned in this article. When a person is tweaking with autoruns, one has to rely on 'trial and error' methods and so there is always the possibility...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |