472,378 Members | 1,678 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,378 software developers and data experts.

Package organization: where to put 'common' modules?

Say I have the following package organization in a system I'm
developing:

A
|----B
|----C
|----D

I have a module, say 'foo', that both package D and B require. What is
the best practice in terms of creating a 'common' package that hosts
'foo'? I want to be able to

- Testing modules in D right in D's directory and testing modules in B
in B's directory;
- If possible, minimize the modification to PYTHONPATH or sys.path
while doing the above.

Is it possible?

Thanks!

Mar 3 '06 #1
11 3716
Ok I guess a little more "hunch" might be needed to elicit suggestions.
If I chose to put foo in A, is there a way for code in B to import foo
from A? Namely, is there a way to import stuff from a parent
directory/package?

If it's not possible without changing sys.path, what's the path of
least effort - i.e., the way that requires least amount of added code
(adding code to prepend sys.path in every module under B certain does
the trick but requires a lot code).

Thanks!

Mar 4 '06 #2
fortepianissimo wrote:
Say I have the following package organization in a system I'm
developing:

A
|----B
|----C
|----D

I have a module, say 'foo', that both package D and B require. What is
the best practice in terms of creating a 'common' package that hosts
'foo'? I want to be able to

- Testing modules in D right in D's directory and testing modules in B
in B's directory;
- If possible, minimize the modification to PYTHONPATH or sys.path
while doing the above.


What I do is run always from the base directory (violates your first
requirement). I make a util package to hold commonly used code. Then B
and D both use
from util import foo

In Python 2.5 you will be able to say (in D, for example)
from ..util import foo

http://www.python.org/peps/pep-0328.html

Kent
Mar 4 '06 #3
Kent Johnson <ke**@kentsjohnson.com> writes:
What I do is run always from the base directory (violates your first
requirement). I make a util package to hold commonly used code. Then B and D
both use
from util import foo

In Python 2.5 you will be able to say (in D, for example)
from ..util import foo

http://www.python.org/peps/pep-0328.html


I do work a bit different here. When two programs start sharing code, then I
convert this code into a "library", i.e., I make it a module, put it in
PYTHONPATH and then import it. Probably more than two projects will use it if
two of them already are :-)
--
Jorge Godoy <go***@ieee.org>

"Quidquid latine dictum sit, altum sonatur."
- Qualquer coisa dita em latim soa profundo.
- Anything said in Latin sounds smart.
Mar 4 '06 #4
Hm this doesn't work. Say I have the following directory structure:

A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py
And bar.py has this line

from util import foo
I then run

python B/bar.py

in directory A. Still got error

ImportError: No module named util
A check of sys.path in bar.py when running from A revealed that
sys.path has A/B, not A.

Am I missing something here?

Mar 5 '06 #5
fortepianissimo wrote:
Hm this doesn't work. Say I have the following directory structure:
A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py

And bar.py has this line

from util import foo

I then run

python B/bar.py

in directory A. Still got error

ImportError: No module named util

Do you have a file util/__init__.py? This is required to make python
recognize util as a package.

This works for me:
C:\WUTemp\A>dir /b/s
C:\WUTemp\A\B
C:\WUTemp\A\util
C:\WUTemp\A\B\bar.py
C:\WUTemp\A\B\__init__.py
C:\WUTemp\A\util\foo.py
C:\WUTemp\A\util\foo.pyc
C:\WUTemp\A\util\__init__.py
C:\WUTemp\A\util\__init__.pyc

C:\WUTemp\A>type util\foo.py
def baz():
print 'foo.baz() here'

C:\WUTemp\A>type B\bar.py
import sys
print sys.path

from util import foo
foo.baz()

C:\WUTemp\A>python B\bar.py
['C:\\WUTemp\\A\\B', 'C:\\Python24\\python24.zip', 'C:\\WUTemp\\A',
<snip lots more dirs>]
foo.baz() here

Kent
Mar 6 '06 #6
fortepianissimo wrote:
Hm this doesn't work. Say I have the following directory structure:

A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py
And bar.py has this line

from util import foo
This would only work with A in sys.path/PYTHONPATH. However...
I then run

python B/bar.py

in directory A. Still got error

ImportError: No module named util
Yes, Python does this - it puts the directory of bar.py (B in this
case) in sys.path, but not the directory in which you're sitting when
you run the program from the shell (A in this case). I always get round
this by running programs like this:

PYTHONPATH=. python B/bar.py

For me, this is a minor inconvenience since I know that ultimately the
imported packages will be installed and available via site-packages.
A check of sys.path in bar.py when running from A revealed that
sys.path has A/B, not A.

Am I missing something here?


Well, I imagine that you want some kind of sys.path modification to
take place, and you can certainly do this in bar.py, taking care only
to do so in the "main program" section of the module - you don't
usually want sys.path magic going on when just importing from the bar
module.

However, my advice with regard to package structure and testing would
be to put common functionality either in modules below those that need
such functionality in the package structure, or in a separate hierarchy
in the package structure, to use absolute references to such packages
(A.util.foo) rather than relative ones (util.foo or stuff soon to be
permitted in Python 2.5), and to put your test programs outside the
package and have them use the package code like any other piece of
software.

Despite the presumed benefits to the upcoming relative import support
(less typing, perhaps), it's good to see that the existing import rules
will be tightened up with absolute importing being enforced. Here's an
example of import misbehaviour in today's Python:

---x
|---y ...print "x"
|---z ...import y.a # we want this to be absolute!

---y
|---a ...print "y"

Now try and import x.z in Python 2.4! It won't even get to printing
"y".

Paul

Mar 6 '06 #7
Interesting - Python seems to act differently under Windows then. Here
I'm using Python 2.4 on Mac OS X. With all of the files, directories
and command exactly like yours, it's still not working.

One thing I noticed is that you have this 'C:\\WUTemp\\A' when you
print sys.path in B/bar.py, but mine doesn't.

Mar 6 '06 #8
Paul Boddie wrote:
fortepianissimo wrote:
Hm this doesn't work. Say I have the following directory structure:

A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py
And bar.py has this line

from util import foo
This would only work with A in sys.path/PYTHONPATH. However...
I then run

python B/bar.py

in directory A. Still got error

ImportError: No module named util


Yes, Python does this - it puts the directory of bar.py (B in this
case) in sys.path, but not the directory in which you're sitting when
you run the program from the shell (A in this case). I always get round
this by running programs like this:

PYTHONPATH=. python B/bar.py

For me, this is a minor inconvenience since I know that ultimately the
imported packages will be installed and available via site-packages.
A check of sys.path in bar.py when running from A revealed that
sys.path has A/B, not A.

Am I missing something here?


Well, I imagine that you want some kind of sys.path modification to
take place, and you can certainly do this in bar.py, taking care only
to do so in the "main program" section of the module - you don't
usually want sys.path magic going on when just importing from the bar
module.


Placing code modifying sys.path in main would be a problem if the
utility loading line is at the beginning of the module - error would
occur before sys.path can be modified.

It is especially true for me since I need to load these "common
functionality" not just for testing - they are necessary to implement
the functionality of the module.

Besides, adding code to manipulate sys.path in *every* module that
needs the common functionality module seems to be quite a hassle...
However, my advice with regard to package structure and testing would
be to put common functionality either in modules below those that need
such functionality in the package structure, or in a separate hierarchy
in the package structure, to use absolute references to such packages
(A.util.foo) rather than relative ones (util.foo or stuff soon to be
permitted in Python 2.5), and to put your test programs outside the
package and have them use the package code like any other piece of
software.
I need to import this common functionality not just for testing code,
so it probably doesn't matter if I separate the testing code from the
module itself.

In addition, the "common functionality" is "common" only for the system
I'm building - they are probably not that useful for general purposes.
So placing them in an absolute path (like site packages) doesn't make a
lot of sense.
Despite the presumed benefits to the upcoming relative import support
(less typing, perhaps), it's good to see that the existing import rules
will be tightened up with absolute importing being enforced. Here's an
example of import misbehaviour in today's Python:

---x
|---y ...print "x"
|---z ...import y.a # we want this to be absolute!

---y
|---a ...print "y"

Now try and import x.z in Python 2.4! It won't even get to printing
"y".


Thanks for pointing this out - I can see this would be a problem if not
careful.

Mar 6 '06 #9
Paul Boddie wrote:
Yes, Python does this - it puts the directory of bar.py (B in this
case) in sys.path, but not the directory in which you're sitting when
you run the program from the shell (A in this case).


This seems to be OS dependent. If I put 'print sys.path' at the start of
site.py (which does most of the setup of sys.path), one element of the
path is an empty string. This is expanded by
main() -> removeduppaths() -> makepath() -> os.path.abspath()
to the current working dir.

This is Python 2.4.2 on Win2K.

Kent
Mar 6 '06 #10
fortepianissimo wrote:
Paul Boddie wrote:
fortepianissimo wrote:
Hm this doesn't work. Say I have the following directory structure:

A
|--- util
| |--- foo.py
|
|--- B
|--- bar.py
And bar.py has this line

from util import foo

[...]
Placing code modifying sys.path in main would be a problem if the
utility loading line is at the beginning of the module - error would
occur before sys.path can be modified.
Well, I suppose you could start your module with...

import sys, os
sys.path.append(some_nasty_tricks_with_paths)

Although I don't really recommend this, since such code is going to get
run as the modules get imported, and this could cause some additional
complications if various things then conspire to import the "wrong"
modules somewhere else in your system.
It is especially true for me since I need to load these "common
functionality" not just for testing - they are necessary to implement
the functionality of the module.

Besides, adding code to manipulate sys.path in *every* module that
needs the common functionality module seems to be quite a hassle...
I suppose you're one of the people who may benefit from the relative
import mechanisms in Python 2.5. Myself, I can't see why it's so hard
to write...

from A.util import foo

[...]
I need to import this common functionality not just for testing code,
so it probably doesn't matter if I separate the testing code from the
module itself.
I stopped running modules inside packages directly precisely because of
the behaviour you've seen. I haven't experienced many problems with my
"test from the outside" approach since.
In addition, the "common functionality" is "common" only for the system
I'm building - they are probably not that useful for general purposes.
So placing them in an absolute path (like site packages) doesn't make a
lot of sense.


Right. So it's just a matter of making sure that everything can import
such functionality without either triggering circular imports or mixing
up "internal" modules with "external" ones (as my example showed).

Paul

Mar 6 '06 #11

Kent Johnson wrote:
Paul Boddie wrote:
Yes, Python does this - it puts the directory of bar.py (B in this
case) in sys.path, but not the directory in which you're sitting when
you run the program from the shell (A in this case).


This seems to be OS dependent. If I put 'print sys.path' at the start of
site.py (which does most of the setup of sys.path), one element of the
path is an empty string. This is expanded by
main() -> removeduppaths() -> makepath() -> os.path.abspath()
to the current working dir.

This is Python 2.4.2 on Win2K.

Kent


Following your example, I tried to use symlink to solve my problem. I
have the following dir structure:

A
|--- util
| |--- foo.py
|
|--- B
| |--- util (symlink to ../util)
| |--- bar.py
|
|--- main.py

------
util/foo.py:

print 'foo initialized'

def baz():
print 'foo.baz() here'

------
B/bar.py:

from util import foo
foo.baz()

------
main.py:

import sys

from util import foo
print id(sys.modules['util.foo'])

from B import bar
print id(sys.modules['util.foo'])
------

Now when I run

python main.py

in dir A, I got the following result (Python 2.4.2, Mac OS X):

foo initialized
3698320
foo initialized
foo.baz() here
3698320
My question is why foo got initialized twice?

Mar 6 '06 #12

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Robert Ferrell | last post by:
I have a question about how to organize modules in a package. I think there is something fundamental that I am missing. Say I'm creating a package, GreatPackage, which has three sub-packages,...
3
by: George P | last post by:
I've run into a strange package related namespace problem. Follow these steps (on UNIX system) to illustrate: ------------------------- mkdir /tmp/mypkg cd /tmp/mypkg touch __init__.py echo...
8
by: Dan Perl | last post by:
Is there a direct way to get a list of all the modules in a package, other than using an os.listdir() for the directory of the package? To be more specific, I would like to inspect every module...
7
by: Sherry Littletree | last post by:
Hi All I am working on a site that has a large amount of common html on all its web pages. I am looking for a way to place this in a single file so, if changes are made, I can change this...
1
by: CLarkou | last post by:
I developed a program in Access. Then I developed add-ins in Excel & Power Point with common modules from Access database. What is the solution for not having to copy+paste code from Access to...
4
by: Fabian Braennstroem | last post by:
Hi, I am pretty new to python and will use it mainly in combination with scientific packages. I am running ubuntu breezy right now and see that some packages are out of date. Do you have any...
49
by: Martin Unsal | last post by:
I'm using Python for what is becoming a sizeable project and I'm already running into problems organizing code and importing packages. I feel like the Python package system, in particular the...
0
by: js | last post by:
Hi list, I'm trying to build a package for python modules. When I just wanted to have a package for Python2.5, this is an easy task, but in most cases, it's not enough. Sometimes I need...
0
by: js | last post by:
By "package", I meant APT, Ports for BSD, MacPorts, etc. On Mon, Jun 16, 2008 at 1:16 AM, Jean-Paul Calderone <exarkun@divmod.comwrote:
0
by: antdb | last post by:
Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was proposed, which integrated multiple engines and...
0
Oralloy
by: Oralloy | last post by:
Hello Folks, I am trying to hook up a CPU which I designed using SystemC to I/O pins on an FPGA. My problem (spelled failure) is with the synthesis of my design into a bitstream, not the C++...
0
by: Carina712 | last post by:
Setting background colors for Excel documents can help to improve the visual appeal of the document and make it easier to read and understand. Background colors can be used to highlight important...
0
BLUEPANDA
by: BLUEPANDA | last post by:
At BluePanda Dev, we're passionate about building high-quality software and sharing our knowledge with the community. That's why we've created a SaaS starter kit that's not only easy to use but also...
1
by: Johno34 | last post by:
I have this click event on my form. It speaks to a Datasheet Subform Private Sub Command260_Click() Dim r As DAO.Recordset Set r = Form_frmABCD.Form.RecordsetClone r.MoveFirst Do If...
1
by: ezappsrUS | last post by:
Hi, I wonder if someone knows where I am going wrong below. I have a continuous form and two labels where only one would be visible depending on the checkbox being checked or not. Below is the...
0
by: jack2019x | last post by:
hello, Is there code or static lib for hook swapchain present? I wanna hook dxgi swapchain present for dx11 and dx9.
0
DizelArs
by: DizelArs | last post by:
Hi all) Faced with a problem, element.click() event doesn't work in Safari browser. Tried various tricks like emulating touch event through a function: let clickEvent = new Event('click', {...
0
by: F22F35 | last post by:
I am a newbie to Access (most programming for that matter). I need help in creating an Access database that keeps the history of each user in a database. For example, a user might have lesson 1 sent...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.