circular imports

[qhfgva]

I'm working with a large code base that I'm slowly trying to fix
"unpythonic" features of.

One feature I'm trying to fix is the use of:

# how things are now
sys.path.append('/general/path/aaa/bbb') # lots of lines like this to
specific dirs
import foo

Insead I'd rather have PYTHONPATH already include '/general/path/'
and then just use:

# how I'd like them to be
from aaa.bbb import foo

So I thought I'd just add the necessary __init__.py files and then
things would just work. Unfortunately trying this exposed a large
number of circular imports which now cause the files to fail to load.

Any ideas why the sys.path.append method has no problem with circular
imports whereas doing thing the "right way" chokes.

thanks,

dustin

[Martin v. Löwis]

qh****@gmail.com wrote:

So I thought I'd just add the necessary __init__.py files and then
things would just work. Unfortunately trying this exposed a large
number of circular imports which now cause the files to fail to load.
You didn't describe you you created the necessary __init__.py files.
If they are not empty, there is a good chance that you made some
error in creating these files.

What is the error that you get with the circular imports?
Any ideas why the sys.path.append method has no problem with circular
imports whereas doing thing the "right way" chokes.

One reason is that there could be duplicate module names, e.g. overlaps
with the standard library. When you fully-qualified the modules, you
actually changed the program, which would now import the user-provided
modules instead of the predefined ones.

Regards,
Martin

[qhfgva]

All of the __init__.py files are empty and I don't know of any
overlapping of names. Like I said this is code that works fine, I'm
just trying to clean up some things as I go. Here are my working
examples:

x1.py
======
# how things work in our code now:
# called with home/dlee/test/module python aaa/x1.py
import sys
sys.path.append('/home/dlee/test/module')

import x2

def goo():
print 'hi from goo'

if __name__ == '__main__':
x2.foo()

x2.py
======
import sys
sys.path.append('/home/dlee/test/module')
import x1

def foo():
print 'entered foo'
x1.goo()
y1.py
======
# this works but is not quite what I want
# called with "PYTHONPATH=$PYTHONPATH:/home/dlee/test/module python
aaa/y1.py"

import aaa.y2

def goo():
print 'hi from goo'

if __name__ == '__main__':
aaa.y2.foo()
y2.py
======
import aaa.y1

def foo():
print 'entered foo'
aaa.y1.goo()
z1.py
======
# how I'd like things to work, but is failing for me
# called with "PYTHONPATH=$PYTHONPATH:/home/dlee/test/module python
aaa/z1.py"

from aaa import z2

def goo():
print 'hi from goo'

if __name__ == '__main__':
z2.foo()
z2.py
======
om aaa import z1

def foo():
print 'entered foo'
z1.goo()


w1.py
======
# this would also be acceptible
# called with "PYTHONPATH=$PYTHONPATH:/home/dlee/test/module python
aaa/w1.py"

import aaa.w2 as w2

def goo():
print 'hi from goo'

if __name__ == '__main__':
w2.foo()
w2.py
======
import aaa.w1 as w1

def foo():
print 'entered foo'
w1.goo()

[Martin v. Löwis]

qh****@gmail.com wrote:

All of the __init__.py files are empty and I don't know of any
overlapping of names. Like I said this is code that works fine, I'm
just trying to clean up some things as I go.

I see. The problem is that a module in a package is entered into
the parent package only after the import is complete. The sequence
goes like this:

create module aaa.z2
add aaa.z2 to sys.modules
run content aaa.z2
add aaa.z2 to aaa

This procedure is needed for rollback in case of exceptions: if
the code in aaa.z2 fails, the traces of the module object must
be discarded. This currently only undoes the sys.modules change.

Now, with circular imports, you get

create module aaa.z2
add aaa.z2 to sys.modules
run aaa.z2
from aaa import z1
is "aaa.z1" already in sys.modules? no - load it
create aaa.z1
add aaa.z1 to sys.modules
run aaa.z1
from aaa import z2
is aaa.z2 already in sys.modules? yes - skip loading it
fetch z2 from aaa -> NameError, aaa has no z2 yet

The problem here is "import from"; this doesn't work with circular
imports too well. For plain "import", circularity is supported, as
the module is added to sys.modules first, then it is run. So you
can import it while its import is in progress.

As a quick work-around, you can add each module to the package:

# z1.py
if __name__ != '__main__':
import sys, aaa
aaa.z1 = sys.modules['aaa.z1']

from aaa import z2

# z2.py
if __name__ != '__main__':
import sys, aaa
aaa.z2 = sys.modules['aaa.z2']

from aaa import z1
HTH,
Martin

P.S. Notice that running a module as __main__ *and* importing it
circularly is also a problem (and is already in the non-package
case): the module gets loaded twice: once as __main__, and once
as z1.

[Peter Hansen]

qh****@gmail.com wrote:

I'm working with a large code base that I'm slowly trying to fix
"unpythonic" features of. [...] Insead I'd rather have PYTHONPATH already include '/general/path/'
and then just use:

One option you might not have considered, which I find more "pythonic"
than environment variables, is to use .pth files as documented at the
top of the site.py module in the standard library (or in the online docs
for "site").

You can also sometimes remove even more problems by using a
sitecustomize.py file in addition to the .pth files. (Also in the docs,
I believe, under "site".)

-Peter

[brian]

One feature I'm trying to fix is the use of:
# how things are now
sys.path.append('/general/path/aaa/bbb') # lots of lines like this to
specific dirs
import foo

I had a need to keep an python source for an application in separate
directories, but not as packages. Rather that manually adjust sys.path
in every single module, I tried to centralize everything:

- config.py in main directory modified sys.path once, adding all the
directories I would need to sys.path.

- sitecustomize.py in every other directory, which did this:
try:
import config
except ImportError:
sys.path.append('/to/main/dir')
import config
(typed from memory, so hopefully no errors)

Hope that helps.
Brian.