Package organization: where to put 'common' modules?

[fortepianissimo]

Say I have the following package organization in a system I'm
developing:

A
|----B
|----C
|----D

I have a module, say 'foo', that both package D and B require. What is
the best practice in terms of creating a 'common' package that hosts
'foo'? I want to be able to

- Testing modules in D right in D's directory and testing modules in B
in B's directory;
- If possible, minimize the modification to PYTHONPATH or sys.path
while doing the above.

Is it possible?

Thanks!

[fortepianissimo]

Ok I guess a little more "hunch" might be needed to elicit suggestions.
If I chose to put foo in A, is there a way for code in B to import foo
from A? Namely, is there a way to import stuff from a parent
directory/package?

If it's not possible without changing sys.path, what's the path of
least effort - i.e., the way that requires least amount of added code
(adding code to prepend sys.path in every module under B certain does
the trick but requires a lot code).

Thanks!

[Kent Johnson]

fortepianissimo wrote:

Say I have the following package organization in a system I'm
developing:

A
|----B
|----C
|----D

I have a module, say 'foo', that both package D and B require. What is
the best practice in terms of creating a 'common' package that hosts
'foo'? I want to be able to

- Testing modules in D right in D's directory and testing modules in B
in B's directory;
- If possible, minimize the modification to PYTHONPATH or sys.path
while doing the above.

What I do is run always from the base directory (violates your first
requirement). I make a util package to hold commonly used code. Then B
and D both use
from util import foo

In Python 2.5 you will be able to say (in D, for example)
from ..util import foo

http://www.python.org/peps/pep-0328.html

Kent

[Jorge Godoy]

Kent Johnson <ke**@kentsjohnson.com> writes:

What I do is run always from the base directory (violates your first
requirement). I make a util package to hold commonly used code. Then B and D
both use
from util import foo

In Python 2.5 you will be able to say (in D, for example)
from ..util import foo

http://www.python.org/peps/pep-0328.html

I do work a bit different here. When two programs start sharing code, then I
convert this code into a "library", i.e., I make it a module, put it in
PYTHONPATH and then import it. Probably more than two projects will use it if
two of them already are :-)
--
Jorge Godoy <go***@ieee.org>

"Quidquid latine dictum sit, altum sonatur."
- Qualquer coisa dita em latim soa profundo.
- Anything said in Latin sounds smart.

[fortepianissimo]

Hm this doesn't work. Say I have the following directory structure:

A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py
And bar.py has this line

from util import foo
I then run

python B/bar.py

in directory A. Still got error

ImportError: No module named util
A check of sys.path in bar.py when running from A revealed that
sys.path has A/B, not A.

Am I missing something here?

[Kent Johnson]

fortepianissimo wrote:

Hm this doesn't work. Say I have the following directory structure:
A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py

And bar.py has this line

from util import foo

I then run

python B/bar.py

in directory A. Still got error

ImportError: No module named util

Do you have a file util/__init__.py? This is required to make python
recognize util as a package.

This works for me:
C:\WUTemp\A>dir /b/s
C:\WUTemp\A\B
C:\WUTemp\A\util
C:\WUTemp\A\B\bar.py
C:\WUTemp\A\B\__init__.py
C:\WUTemp\A\util\foo.py
C:\WUTemp\A\util\foo.pyc
C:\WUTemp\A\util\__init__.py
C:\WUTemp\A\util\__init__.pyc

C:\WUTemp\A>type util\foo.py
def baz():
print 'foo.baz() here'

C:\WUTemp\A>type B\bar.py
import sys
print sys.path

from util import foo
foo.baz()

C:\WUTemp\A>python B\bar.py
['C:\\WUTemp\\A\\B', 'C:\\Python24\\python24.zip', 'C:\\WUTemp\\A',
<snip lots more dirs>]
foo.baz() here

Kent

[Paul Boddie]

fortepianissimo wrote:

Hm this doesn't work. Say I have the following directory structure:

A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py
And bar.py has this line

from util import foo
This would only work with A in sys.path/PYTHONPATH. However...
I then run

python B/bar.py

in directory A. Still got error

ImportError: No module named util
Yes, Python does this - it puts the directory of bar.py (B in this
case) in sys.path, but not the directory in which you're sitting when
you run the program from the shell (A in this case). I always get round
this by running programs like this:

PYTHONPATH=. python B/bar.py

For me, this is a minor inconvenience since I know that ultimately the
imported packages will be installed and available via site-packages.
A check of sys.path in bar.py when running from A revealed that
sys.path has A/B, not A.

Am I missing something here?

Well, I imagine that you want some kind of sys.path modification to
take place, and you can certainly do this in bar.py, taking care only
to do so in the "main program" section of the module - you don't
usually want sys.path magic going on when just importing from the bar
module.

However, my advice with regard to package structure and testing would
be to put common functionality either in modules below those that need
such functionality in the package structure, or in a separate hierarchy
in the package structure, to use absolute references to such packages
(A.util.foo) rather than relative ones (util.foo or stuff soon to be
permitted in Python 2.5), and to put your test programs outside the
package and have them use the package code like any other piece of
software.

Despite the presumed benefits to the upcoming relative import support
(less typing, perhaps), it's good to see that the existing import rules
will be tightened up with absolute importing being enforced. Here's an
example of import misbehaviour in today's Python:

---x
|---y ...print "x"
|---z ...import y.a # we want this to be absolute!

---y
|---a ...print "y"

Now try and import x.z in Python 2.4! It won't even get to printing
"y".

Paul

[fortepianissimo]

Interesting - Python seems to act differently under Windows then. Here
I'm using Python 2.4 on Mac OS X. With all of the files, directories
and command exactly like yours, it's still not working.

One thing I noticed is that you have this 'C:\\WUTemp\\A' when you
print sys.path in B/bar.py, but mine doesn't.

[fortepianissimo]

Paul Boddie wrote:

fortepianissimo wrote:

Hm this doesn't work. Say I have the following directory structure:

A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py
And bar.py has this line

from util import foo
This would only work with A in sys.path/PYTHONPATH. However...

I then run

python B/bar.py

in directory A. Still got error

ImportError: No module named util

Yes, Python does this - it puts the directory of bar.py (B in this
case) in sys.path, but not the directory in which you're sitting when
you run the program from the shell (A in this case). I always get round
this by running programs like this:

PYTHONPATH=. python B/bar.py

For me, this is a minor inconvenience since I know that ultimately the
imported packages will be installed and available via site-packages.

A check of sys.path in bar.py when running from A revealed that
sys.path has A/B, not A.

Am I missing something here?

Well, I imagine that you want some kind of sys.path modification to
take place, and you can certainly do this in bar.py, taking care only
to do so in the "main program" section of the module - you don't
usually want sys.path magic going on when just importing from the bar
module.

Placing code modifying sys.path in main would be a problem if the
utility loading line is at the beginning of the module - error would
occur before sys.path can be modified.

It is especially true for me since I need to load these "common
functionality" not just for testing - they are necessary to implement
the functionality of the module.

Besides, adding code to manipulate sys.path in *every* module that
needs the common functionality module seems to be quite a hassle...
However, my advice with regard to package structure and testing would
be to put common functionality either in modules below those that need
such functionality in the package structure, or in a separate hierarchy
in the package structure, to use absolute references to such packages
(A.util.foo) rather than relative ones (util.foo or stuff soon to be
permitted in Python 2.5), and to put your test programs outside the
package and have them use the package code like any other piece of
software.
I need to import this common functionality not just for testing code,
so it probably doesn't matter if I separate the testing code from the
module itself.

In addition, the "common functionality" is "common" only for the system
I'm building - they are probably not that useful for general purposes.
So placing them in an absolute path (like site packages) doesn't make a
lot of sense.
Despite the presumed benefits to the upcoming relative import support
(less typing, perhaps), it's good to see that the existing import rules
will be tightened up with absolute importing being enforced. Here's an
example of import misbehaviour in today's Python:

---x
|---y ...print "x"
|---z ...import y.a # we want this to be absolute!

---y
|---a ...print "y"

Now try and import x.z in Python 2.4! It won't even get to printing
"y".

Thanks for pointing this out - I can see this would be a problem if not
careful.

[Kent Johnson]

Paul Boddie wrote:

Yes, Python does this - it puts the directory of bar.py (B in this
case) in sys.path, but not the directory in which you're sitting when
you run the program from the shell (A in this case).

This seems to be OS dependent. If I put 'print sys.path' at the start of
site.py (which does most of the setup of sys.path), one element of the
path is an empty string. This is expanded by
main() -> removeduppaths() -> makepath() -> os.path.abspath()
to the current working dir.

This is Python 2.4.2 on Win2K.

Kent

[Paul Boddie]

fortepianissimo wrote:

Paul Boddie wrote:

fortepianissimo wrote:

Hm this doesn't work. Say I have the following directory structure:

A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py
And bar.py has this line

from util import foo

[...]
Placing code modifying sys.path in main would be a problem if the
utility loading line is at the beginning of the module - error would
occur before sys.path can be modified.
Well, I suppose you could start your module with...

import sys, os
sys.path.append(some_nasty_tricks_with_paths)

Although I don't really recommend this, since such code is going to get
run as the modules get imported, and this could cause some additional
complications if various things then conspire to import the "wrong"
modules somewhere else in your system.
It is especially true for me since I need to load these "common
functionality" not just for testing - they are necessary to implement
the functionality of the module.

Besides, adding code to manipulate sys.path in *every* module that
needs the common functionality module seems to be quite a hassle...
I suppose you're one of the people who may benefit from the relative
import mechanisms in Python 2.5. Myself, I can't see why it's so hard
to write...

from A.util import foo

[...]
I need to import this common functionality not just for testing code,
so it probably doesn't matter if I separate the testing code from the
module itself.
I stopped running modules inside packages directly precisely because of
the behaviour you've seen. I haven't experienced many problems with my
"test from the outside" approach since.
In addition, the "common functionality" is "common" only for the system
I'm building - they are probably not that useful for general purposes.
So placing them in an absolute path (like site packages) doesn't make a
lot of sense.

Right. So it's just a matter of making sure that everything can import
such functionality without either triggering circular imports or mixing
up "internal" modules with "external" ones (as my example showed).

Paul

[fortepianissimo]

Kent Johnson wrote:

Paul Boddie wrote:

Yes, Python does this - it puts the directory of bar.py (B in this
case) in sys.path, but not the directory in which you're sitting when
you run the program from the shell (A in this case).

This seems to be OS dependent. If I put 'print sys.path' at the start of
site.py (which does most of the setup of sys.path), one element of the
path is an empty string. This is expanded by
main() -> removeduppaths() -> makepath() -> os.path.abspath()
to the current working dir.

This is Python 2.4.2 on Win2K.

Kent

Following your example, I tried to use symlink to solve my problem. I
have the following dir structure:

A
|--- util
| |--- foo.py
|
|--- B
| |--- util (symlink to ../util)
| |--- bar.py
|
|--- main.py

------
util/foo.py:

print 'foo initialized'

def baz():
print 'foo.baz() here'

------
B/bar.py:

from util import foo
foo.baz()

------
main.py:

import sys

from util import foo
print id(sys.modules['util.foo'])

from B import bar
print id(sys.modules['util.foo'])
------

Now when I run

python main.py

in dir A, I got the following result (Python 2.4.2, Mac OS X):

foo initialized
3698320
foo initialized
foo.baz() here
3698320
My question is why foo got initialized twice?