Insane import behaviour and regrtest.py: heeelp

[John J. Lee]

I'm tearing my hair out at what seems like weird import behaviour I'm
getting from Python's stdlib test script, regrtest.py (not for the
first time: seem to have forgotten the resolution from last time, and
the time before, and the time before that, when this damn test script
of Python's had me screaming at the screen... I don't seem to be
learning my lesson here :( )

*Please* somebody inform me *what* affects what file Python will pick
up, other than:

0. Python executable

1. sys.path

2. current working directory

3. Directory in which script lives

4. Directory from which script was invoked

5. The contents of one's hard drive

Even with all of these constant (and not all of them should directly
affect the outcome in any case, I know), I get different modules
picked up from imports in two scripts, and I'm damned if I can see
why.

Here's the first script, just a test script:

import os, sys
sys.path = ['/hda/usr/local/buf/python/python/dist/src/Lib', '/home/john/lib/python', '/usr/local/lib/python24.zip', '/usr/local/lib/python2.4', '/usr/local/lib/python2.4/plat-linux2', '/usr/local/lib/python2.4/lib-tk', '/usr/local/lib/python2.4/lib-dynload', '/usr/local/lib/python2.4/site-packages']
abstest = "test.test_urllib2"
print 'abstest', abstest
print 'sys.path', sys.path
print 'cwd', os.getcwd()
the_package = __import__(abstest, globals(), locals(), [])
print 'the_package.__path__', the_package.__path__

If I stick that script in my Python CVS Lib/test directory as blah.py,
then cd to Lib and:

$ python2.4 test/blah.py
abstest test.test_urllib2
sys.path ['/hda/usr/local/buf/python/python/dist/src/Lib', '/home/john/lib/python', '/usr/local/lib/python24.zip', '/usr/local/lib/python2.4', '/usr/local/lib/python2.4/plat-linux2', '/usr/local/lib/python2.4/lib-tk', '/usr/local/lib/python2.4/lib-dynload', '/usr/local/lib/python2.4/site-packages']
cwd /hda/usr/local/buf/python/python/dist/src/Lib
*************
the_package.__path__ ['/hda/usr/local/buf/python/python/dist/src/Lib/test']
All as expected: I pick up Lib/test/test_urllib2.py -- that *******
business is coming from a print >> sys.stderr statement I inserted at
module level in that copy of the module (though the >> sys.stderr is
probably unnecessary, since IIRC regrtest does not rebind sys.stdout
when the -v option is given, as in my usage below).
But, when I run the second script, Lib/test/regrtest.py, with a
modified sys.path (see paragraph below) and the same print statements
as in my test script inserted:

$ python2.4 test/regrtest.py -f test/torun -v
test_urllib2
abstest test.test_urllib2
sys.path ['/hda/usr/local/buf/python/python/dist/src/Lib', '/home/john/lib/python', '/usr/local/lib/python24.zip', '/usr/local/lib/python2.4', '/usr/local/lib/python2.4/plat-linux2', '/usr/local/lib/python2.4/lib-tk', '/usr/local/lib/python2.4/lib-dynload', '/usr/local/lib/python2.4/site-packages']
cwd /hda/usr/local/buf/python/python/dist/src/Lib
the_package.__path__ ['/usr/local/lib/python2.4/test']

I get the test_urllib2.py from /usr/local/lib/python2.4 (note __path__
and lack of *****s), though everything else (sys.path, cwd, etc.)
remains the same as my test script above!

The only change I made to regrtest other than the print statements was
to add Lib to sys.path, so I pick up modules from CVS instead of the
installed 2.4 versions (yeah, I know I should build and install a
python2.5 executable from CVS, but I don't see how that's relevant here).

Why??? Help!

stupid-stupid-stupid-gnnnnhhhhh-<wink>-ly y'rs,
John

[Nick Coghlan]

John J. Lee wrote:

The only change I made to regrtest other than the print statements was
to add Lib to sys.path, so I pick up modules from CVS instead of the
installed 2.4 versions (yeah, I know I should build and install a
python2.5 executable from CVS, but I don't see how that's relevant here).

I didn't read your whole message so I may be off track here, but regrtest.py has
some arguably demented path handling behaviour, and it's driven mainly by the
physical location of regrtest.py. In particular:

def findtestdir():
if __name__ == '__main__':
file = sys.argv[0]
else:
file = __file__
testdir = os.path.dirname(file) or os.curdir
return testdir

So intead of adding anything to sys.path, tweak the call to 'main' on the final
line to set "testdir" appropriately.

Cheers,
Nick.

--
Nick Coghlan | nc******@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

[John J. Lee]

Nick Coghlan <nc******@iinet.net.au> writes:

John J. Lee wrote:

The only change I made to regrtest other than the print statements was
to add Lib to sys.path, so I pick up modules from CVS instead of the
installed 2.4 versions (yeah, I know I should build and install a
python2.5 executable from CVS, but I don't see how that's relevant here).

I didn't read your whole message so I may be off track here, but
regrtest.py has some arguably demented path handling behaviour, and
it's driven mainly by the physical location of regrtest.py. In
particular:

def findtestdir():
if __name__ == '__main__':
file = sys.argv[0]
else:
file = __file__
testdir = os.path.dirname(file) or os.curdir
return testdir

So intead of adding anything to sys.path, tweak the call to 'main' on
the final line to set "testdir" appropriately.

Nope: that just affects finding the *names* of the tests. It does not
determine the location from which they are actually imported. I tried
it, and I get the same results as before (the test modules from my
installed copy of Python are picked up instead of the local copies in
my CVS checkout's Lib/test, apparently entirely contrary to sys.path).
John

[Tim Peters]

[John J. Lee]

...
I tried it, and I get the same results as before (the test modules from my
installed copy of Python are picked up instead of the local copies in
my CVS checkout's Lib/test, apparently entirely contrary to sys.path).

Try running Python with -vv (that's two letter "v", not one letter
"w"). That will display a long-winded account of everything Python
tries in order to satisfy an import.

[John J. Lee]

Tim Peters <ti********@gmail.com> writes:

[John J. Lee]

...
I tried it, and I get the same results as before (the test modules from my
installed copy of Python are picked up instead of the local copies in
my CVS checkout's Lib/test, apparently entirely contrary to sys.path).

Try running Python with -vv (that's two letter "v", not one letter
"w"). That will display a long-winded account of everything Python
tries in order to satisfy an import.

Thanks.

I'm still puzzled, though. Reading the -vv output, I see that when
importing test_cookielib (which is currently the only line in my
regrtest.py -f file), Python doesn't appear to look in Lib/test to
find module "test.test_cookielib" (the actual string passed by
regrtest.py to __import__): see the output below.

Why does Python not look in

/usr/local/src/python/python/dist/src/Lib/test

for test_cookielib.py{,c,o}??
Snippet to show print statements I added to my regrtest.py :

.. try:
.. save_stdout = sys.stdout
.. try:
.. if cfp:
.. sys.stdout = cfp
.. print test # Output file starts with test name
.. if test.startswith('test.'):
.. abstest = test
.. else:
.. # Always import it from the test package
.. abstest = 'test.' + test
..+ print >> sys.stderr, ">>>>>>>sys.path", sys.path
.. the_package = __import__(abstest, globals(), locals(), [])
..+ print >> sys.stderr, ">>>>>>>the_package.__path__", the_package.__path__
.. the_module = getattr(the_package, test)
.. # Most tests run to completion simply as a side-effect of

Only other change to regrtest.py:

.. if __name__ == '__main__':
.. # Remove regrtest.py's own directory from the module search path. This
.. # prevents relative imports from working, and relative imports will screw
.. # up the testing framework. E.g. if both test.test_support and
.. # test_support are imported, they will not contain the same globals, and
.. # much of the testing framework relies on the globals in the
.. # test.test_support module.
.. mydir = os.path.abspath(os.path.normpath(os.path.dirname(s ys.argv[0])))
.. i = pathlen = len(sys.path)
.. while i >= 0:
.. i -= 1
.. if os.path.abspath(os.path.normpath(sys.path[i])) == mydir:
.. del sys.path[i]
.. if len(sys.path) == pathlen:
.. print 'Could not find %r in sys.path to remove it' % mydir
..+ lib = "/usr/local/src/python/python/dist/src/Lib"
..+ sys.path.insert(0, lib)
..- main()
..+ main(testdir=lib)
Here's what I've got in my CVS checkout:

Lib[0]$ pwd
/hda/usr/local/buf/python/python/dist/src/Lib
Lib[0]$ ls test/test_cookielib* | grep -v 'patch\|latest\|~'
test/test_cookielib.py
test/test_cookielib.pyc
And the output:

Lib[0]$ python2.4 -u -vv test/regrtest.py -f test/torun -v 2>&1 | less
[...long snip...]
# trying /usr/local/lib/python2.4/lib-tk/time.py
# trying /usr/local/lib/python2.4/lib-tk/time.pyc
# trying /usr/local/lib/python2.4/lib-dynload/time.so
dlopen("/usr/local/lib/python2.4/lib-dynload/time.so", 2);
import time # dynamically loaded from /usr/local/lib/python2.4/lib-dynload/time.so
test_cookielib

>>>sys.path ['/usr/local/src/python/python/dist/src/Lib', '/home/john/lib/python', '/usr/local/lib/python24.zip', '/usr/local/lib/python2.4', '/usr/local/lib/python2.4/plat-linux2', '/usr/local/lib/python2.4/lib-tk', '/usr/local/lib/python2.4/lib-dynload', '/usr/local/lib/python2.4/site-packages'] # trying /usr/local/lib/python2.4/test/test_cookielib.so
# trying /usr/local/lib/python2.4/test/test_cookielibmodule.so
# trying /usr/local/lib/python2.4/test/test_cookielib.py
import test.test_cookielib # from /usr/local/lib/python2.4/test/test_cookielib.py
# can't create /usr/local/lib/python2.4/test/test_cookielib.pyc
# trying /usr/local/lib/python2.4/test/re.so
# trying /usr/local/lib/python2.4/test/remodule.so
# trying /usr/local/lib/python2.4/test/re.py
# trying /usr/local/lib/python2.4/test/re.pyc
# trying /usr/local/lib/python2.4/test/time.so
# trying /usr/local/lib/python2.4/test/timemodule.so
# trying /usr/local/lib/python2.4/test/time.py
# trying /usr/local/lib/python2.4/test/time.pyc
# trying /usr/local/lib/python2.4/test/test.so
# trying /usr/local/lib/python2.4/test/testmodule.so
# trying /usr/local/lib/python2.4/test/test.py
# trying /usr/local/lib/python2.4/test/test.pyc>>>the_package.__path__ ['/usr/local/lib/python2.4/test']

# trying /usr/local/lib/python2.4/test/test_sets.so
# trying /usr/local/lib/python2.4/test/test_setsmodule.so
# trying /usr/local/lib/python2.4/test/test_sets.py
# /usr/local/lib/python2.4/test/test_sets.pyc matches /usr/local/lib/python2.4/test/test_sets.py
import test.test_sets # precompiled from /usr/local/lib/python2.4/test/test_sets.pyc
# trying /usr/local/lib/python2.4/test/operator.so
# trying /usr/local/lib/python2.4/test/operatormodule.so
# trying /usr/local/lib/python2.4/test/operator.py
# trying /usr/local/lib/python2.4/test/operator.pyc
# trying /usr/local/src/python/python/dist/src/Lib/operator.so
[...long snip...]
John

[Tim Peters]

[John J. Lee]

I'm still puzzled, though. Reading the -vv output, I see that when
importing test_cookielib (which is currently the only line in my
regrtest.py -f file), Python doesn't appear to look in Lib/test to
find module "test.test_cookielib" (the actual string passed by
regrtest.py to __import__): see the output below.

Why does Python not look in

/usr/local/src/python/python/dist/src/Lib/test

for test_cookielib.py{,c,o}??
Can't guess from here, from what you've shown. Obvious possibilities:

1. /usr/local/src/python/python/dist/src/Lib
isn't on sys.path

2. /usr/local/src/python/python/dist/src/Lib
is on sys.path but directory
/usr/local/src/python/python/dist/src/Lib/test
doesn't exist

3. /usr/local/src/python/python/dist/src/Lib
is on sys.path and directory
/usr/local/src/python/python/dist/src/Lib/test
exists but one of these files doesn't exist:
a. /usr/local/src/python/python/dist/src/Lib/test/test_cookielib.py
b. /usr/local/src/python/python/dist/src/Lib/test/__init__.py
Snippet to show print statements I added to my regrtest.py :

. try:
. save_stdout = sys.stdout
. try:
. if cfp:
. sys.stdout = cfp
. print test # Output file starts with test name
. if test.startswith('test.'):
. abstest = test
. else:
. # Always import it from the test package
. abstest = 'test.' + test
.+ print >> sys.stderr, ">>>>>>>sys.path", sys.path
. the_package = __import__(abstest, globals(), locals(), [])
.+ print >> sys.stderr, ">>>>>>>the_package.__path__", the_package.__path__
. the_module = getattr(the_package, test)
. # Most tests run to completion simply as a side-effect of

Only other change to regrtest.py:

. if __name__ == '__main__':
. # Remove regrtest.py's own directory from the module search path. This
. # prevents relative imports from working, and relative imports will screw
. # up the testing framework. E.g. if both test.test_support and
. # test_support are imported, they will not contain the same globals, and
. # much of the testing framework relies on the globals in the
. # test.test_support module.
. mydir = os.path.abspath(os.path.normpath(os.path.dirname(s ys.argv[0])))
. i = pathlen = len(sys.path)
. while i >= 0:
. i -= 1
. if os.path.abspath(os.path.normpath(sys.path[i])) == mydir:
. del sys.path[i]
. if len(sys.path) == pathlen:
. print 'Could not find %r in sys.path to remove it' % mydir
.+ lib = "/usr/local/src/python/python/dist/src/Lib"
.+ sys.path.insert(0, lib)
.- main()
.+ main(testdir=lib)

Here's what I've got in my CVS checkout:

Lib[0]$ pwd
/hda/usr/local/buf/python/python/dist/src/Lib
That doesn't look to be the same thing as the

/usr/local/src/python/python/dist/src/Lib

you're asking about, right?
Lib[0]$ ls test/test_cookielib* | grep -v 'patch\|latest\|~'
test/test_cookielib.py
test/test_cookielib.pyc
Since those files are apparenlty

/hda/usr/local/buf/python/python/dist/src/Lib/test/test_cookielib.py
/hda/usr/local/buf/python/python/dist/src/Lib/test/test_cookielib.pyc

and

/hda/usr/local/buf/python/python/dist/src/Lib

isn't in your sys.path:

['/usr/local/src/python/python/dist/src/Lib',
'/home/john/lib/python',
'/usr/local/lib/python24.zip',
'/usr/local/lib/python2.4',
'/usr/local/lib/python2.4/plat-linux2',
'/usr/local/lib/python2.4/lib-tk',
'/usr/local/lib/python2.4/lib-dynload',
'/usr/local/lib/python2.4/site-packages'
]

I don't understand why you think that "ls" output is (or should be) relevant.
And the output:

Lib[0]$ python2.4 -u -vv test/regrtest.py -f test/torun -v 2>&1 | less
[...long snip...]
test_cookielib

>>>>sys.path ['/usr/local/src/python/python/dist/src/Lib', '/home/john/lib/python', '/usr/local/lib/python24.zip', '/usr/local/lib/python2.4', '/usr/local/lib/python2.4/plat-linux2', '/usr/local/lib/python2.4/lib-tk', '/usr/local/lib/python2.4/lib-dynload', '/usr/local/lib/python2.4/site-packages']

# trying /usr/local/lib/python2.4/test/test_cookielib.so
# trying /usr/local/lib/python2.4/test/test_cookielibmodule.so
# trying /usr/local/lib/python2.4/test/test_cookielib.py
import test.test_cookielib # from /usr/local/lib/python2.4/test/test_cookielib.py
# can't create /usr/local/lib/python2.4/test/test_cookielib.pyc

....

See #2 and #3 above for some obvious possiblities not ruled out by
anything shown here. I just don't see any mystery here, but might if
both of these displayed something interesting:

$ ls -l /usr/local/src/python/python/dist/src/Lib/test/test_cookielib.py
$ ls -l /usr/local/src/python/python/dist/src/Lib/test/__init__.py

[John J. Lee]

Tim Peters <ti********@gmail.com> writes:

[John J. Lee]

I'm still puzzled, though. Reading the -vv output, I see that when [...] Lib[0]$ pwd
/hda/usr/local/buf/python/python/dist/src/Lib

That doesn't look to be the same thing as the

/usr/local/src/python/python/dist/src/Lib

you're asking about, right?

[...]

Ah: somehow managed to look straight through that, so used am I to the
presence of these symlinks:

Lib[0]$ ls -l /usr/local/src
lrwxrwxrwx 1 root root 19 Sep 4 13:45 /usr/local/src -> /hda/usr/local/src//
Lib[0]$ ls -l /hda/usr/local/src/python
lrwxr-xr-x 1 john oldjohn 25 Sep 11 13:21 /hda/usr/local/src/python -> /hda/usr/local/buf/python/
The twisty mess of symlinks and owners is down to a couple of OS and
hard disk changes.

I don't immediately see why at 1am, but I guess that's the problem
(that, and trying to understand import details at 1am...).

Thanks!
John