python24.zip

[Robin Becker]

Investigating a query about the python path I see that my win32 installation has
c:/windows/system32/python24.zip (which is non existent) second on sys.path
before the actual python24/lib etc etc.

Firstly should python start up with non-existent entries on the path?
Secondly is this entry be the default for some other kind of python installation?
--
Robin Becker

[Martin v. Löwis]

Robin Becker wrote:

Firstly should python start up with non-existent entries on the path?
Yes, this is by design.
Secondly is this entry be the default for some other kind of python
installation?

Yes. People can package everything they want in python24.zip (including
site.py). This can only work if python24.zip is already on the path
(and I believe it will always be sought in the directory where
python24.dll lives).

Regards,
Martin

[Dieter Maurer]

"Martin v. Löwis" <ma****@v.loewi s.de> writes on Fri, 20 May 2005 18:13:56 +0200:

Robin Becker wrote:

Firstly should python start up with non-existent entries on the path?

Yes, this is by design.

Secondly is this entry be the default for some other kind of python
installation?

Yes. People can package everything they want in python24.zip (including
site.py). This can only work if python24.zip is already on the path
(and I believe it will always be sought in the directory where
python24.dll lives).

The question was:

"should python start up with **non-existent** objects on the path".

I think there is no reason why path needs to contain an object
which does not exist (at the time the interpreter starts).

In your use case, "python24.z ip" does exist and therefore may
be on the path. When "python24.z ip" does not exist, it does
not contain anything and especially not "site.py".
I recently analysed excessive import times and
saw thousands of costly and unneccesary filesystem operations due to:

* long "sys.path", especially containing non-existing objects

Although non-existent, about 5 filesystem operations are
tried on them for any module not yet located.

* a severe weakness in Python's import hook treatment

When there is an importer "i" for a path "p" and
this importer cannot find module "m", then "p" is
treated as a directory and 5 file system operations
are tried to locate "p/m". Of course, all of them fail
when "p" happens to be a zip archive.
Dieter

[Robin Becker]

Dieter Maurer wrote:
......

The question was:

"should python start up with **non-existent** objects on the path".

I think there is no reason why path needs to contain an object
which does not exist (at the time the interpreter starts).

In your use case, "python24.z ip" does exist and therefore may
be on the path. When "python24.z ip" does not exist, it does
not contain anything and especially not "site.py".

I think this was my intention, but also I think I have some concern over
having two possible locations for the standard library. It seems non pythonic
and liable to cause confusion if some package should manage to install
python24.zip while I believe that python24\lib is being used.

I recently analysed excessive import times and
saw thousands of costly and unneccesary filesystem operations due to:

* long "sys.path", especially containing non-existing objects

Although non-existent, about 5 filesystem operations are
tried on them for any module not yet located.

* a severe weakness in Python's import hook treatment

When there is an importer "i" for a path "p" and
this importer cannot find module "m", then "p" is
treated as a directory and 5 file system operations
are tried to locate "p/m". Of course, all of them fail
when "p" happens to be a zip archive.
Dieter

I suppose that's a reason for eliminating duplicates and non-existent entries.

--
Robin Becker

[Martin v. Löwis]

Dieter Maurer wrote:

The question was:

"should python start up with **non-existent** objects on the path".

I think there is no reason why path needs to contain an object
which does not exist (at the time the interpreter starts).
There is. When the interpreter starts, it doesn't know what object
do or do not exist. So it must put python24.zip on the path
just in case.
In your use case, "python24.z ip" does exist and therefore may
be on the path. When "python24.z ip" does not exist, it does
not contain anything and especially not "site.py".
Yes, but the interpreter cannot know in advance whether
python24.zip will be there when it starts.
I recently analysed excessive import times and
saw thousands of costly and unneccesary filesystem operations due to:

Hmm. In my Python 2.4 installation, I only get 154 open calls, and
63 stat calls on an empty Python file. So somebody must have messed
with sys.path really badly if you saw thoughsands of file operations
(although I wonder what operating system you use so that failing
open operations are costly; most operating systems should do them
very efficiently).

Regards,
Martin

[Steve Holden]

Robin Becker wrote:

Dieter Maurer wrote: [...]
I think this was my intention, but also I think I have some concern over
having two possible locations for the standard library. It seems non pythonic
and liable to cause confusion if some package should manage to install
python24.zip while I believe that python24\lib is being used.

I recently analysed excessive import times and
saw thousands of costly and unneccesary filesystem operations due to:

* long "sys.path", especially containing non-existing objects

Although non-existent, about 5 filesystem operations are
tried on them for any module not yet located.

* a severe weakness in Python's import hook treatment

When there is an importer "i" for a path "p" and
this importer cannot find module "m", then "p" is
treated as a directory and 5 file system operations
are tried to locate "p/m". Of course, all of them fail
when "p" happens to be a zip archive.
Dieter

I suppose that's a reason for eliminating duplicates and non-existent entries.

There are some aspects of Python's initialization that are IMHO a bit
too filesystem-dependent. I mentioned one in
http://sourceforge.net/tracker/index...70&atid=105470

but I'd appreciate further support. Ideally there should be some means
for hooked import mechanisms to provide answers that are currently
sought from the filestore.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/

[Dieter Maurer]

"Martin v. Löwis" <ma****@v.loewi s.de> writes on Sat, 21 May 2005 23:53:31 +0200:

Dieter Maurer wrote:
...

The question was:

"should python start up with **non-existent** objects on the path".

I think there is no reason why path needs to contain an object
which does not exist (at the time the interpreter starts).
There is. When the interpreter starts, it doesn't know what object
do or do not exist. So it must put python24.zip on the path
just in case.

Really?

Is the interpreter unable to call "C" functions ("stat" for example)
to determine whether an object exists before it puts it on "path".
Yes, but the interpreter cannot know in advance whether
python24.zip will be there when it starts.

Thus, it checks dynamically when it starts.

I recently analysed excessive import times and
saw thousands of costly and unneccesary filesystem operations due to:

Hmm. In my Python 2.4 installation, I only get 154 open calls, and
63 stat calls on an empty Python file. So somebody must have messed
with sys.path really badly if you saw thoughsands of file operations
(although I wonder what operating system you use so that failing
open operations are costly; most operating systems should do them
very efficiently).

The application was Zope importing about 2.500 modules
from 2 zip files "zope.zip" and "python24.z ip".
This resulted in about 12.500 opens -- about 4 times more
than would be expected -- about 10.000 of them failing opens.
Dieter

[Dieter Maurer]

Steve Holden <st***@holdenwe b.com> writes on Sun, 22 May 2005 09:14:43 -0400:

...
There are some aspects of Python's initialization that are IMHO a bit
too filesystem-dependent. I mentioned one in
http://sourceforge.net/tracker/index...70&atid=105470
but I'd appreciate further support. Ideally there should be some means
for hooked import mechanisms to provide answers that are currently
sought from the filestore.

There are such hooks. See e.g. the "meta_path" hooks as
described by PEP 302.

[Martin v. Löwis]

Dieter Maurer wrote:

Really?

Is the interpreter unable to call "C" functions ("stat" for example)
to determine whether an object exists before it puts it on "path".
What do you mean, "unable to"? It just doesn't.

Could it? Perhaps, if somebody wrote a patch.
Would the patch be accepted? Perhaps, if it didn't break something
else.

In the past, there was a silent guarantee that you could add
items to sys.path, and only later create the directories behind
these items. I don't know whether people rely on this guarantee.
The application was Zope importing about 2.500 modules
from 2 zip files "zope.zip" and "python24.z ip".
This resulted in about 12.500 opens -- about 4 times more
than would be expected -- about 10.000 of them failing opens.

I see. Out of curiosity: how much startup time was saved
when sys.path was explicitly stripped to only contain these
two zip files?

I would expect that importing 2500 modules takes *way*
more time than doing 10.000 failed opens.

Regards,
Martin

[Steve Holden]

Dieter Maurer wrote:

Steve Holden <st***@holdenwe b.com> writes on Sun, 22 May 2005 09:14:43 -0400:

...
There are some aspects of Python's initialization that are IMHO a bit
too filesystem-dependent. I mentioned one in
http://sourceforge.net/tracker/index...70&atid=105470
but I'd appreciate further support. Ideally there should be some means
for hooked import mechanisms to provide answers that are currently
sought from the filestore.

There are such hooks. See e.g. the "meta_path" hooks as
described by PEP 302.

Indeed I have written PEP 302-based code to import from a relational
database, but I still don't believe there's any satisfactory way to have
[such a hooked import mechanism] be a first-class component of an
architecture that specifically requires an os.py to exist in the file
store during initialization.

I wasn't asking for an import hook mechanism (since I already knew these
to exist), but for a way to allow such mechanisms to be the sole import
support for certain implementations .

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/