By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,929 Members | 1,245 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,929 IT Pros & Developers. It's quick & easy.

Code Management

P: n/a
Dear Reader,

I'm writing some modules in Python, and I'm also using unittests. I'm
wondering about some things:

1) Should I put my unittests in a subdirectory? Does the subdirectory
have to be a package?

2) Does the main folder /myproject have to be a package? Should I put
my modules directly under /myproject, or should I create a subfolder,
for example /myproject/modules

Does anyone have any "best practices" as to how to manage your code?

Thanks!
Nov 21 '07 #1
Share this Question
Share on Google+
11 Replies


P: n/a
Let's see:
1) Should I put my unittests in a subdirectory? Does the subdirectory
have to be a package?
Sure, to avoid cluttering put the tests in a folder called 'tests'
(and any input the tests require, like mock files, output files used
to test if the output is correct, etc etc, should be put in that
folder or in a subfolder of 'tests'.

I like this setup:

myproject
.../ package1
../../ tests
.../../../ mock_files
.../package2

and so on

If you want to be able to import the tests from python (say, 'from
myproject.tests import test_all'), then yes, you need to make the
tests folder a subpackage (just drop an empty __init__.py file)
2) Does the main folder /myproject have to be a package? Should I put
my modules directly under /myproject, or should I create a subfolder,
for example /myproject/modules
Of course... make /myproject a package. Now, if your modules are kinda
small you can just dump them in /myproject, but if what you are doing
is fairly complex, it's better to move each module to its own folder.
Does anyone have any "best practices" as to how to manage your code?
I want 'best practices' too... anyone?

Sergio
PS: Casey, neat idea with the 'alltests.py'. Btw, I think relative
imports can be helpful:
http://www.python.org/dev/peps/pep-0...ido-s-decision
Nov 21 '07 #2

P: n/a
Jens <j3****@gmail.comwrites:
On 21 Nov., 04:16, Jens <j3n...@gmail.comwrote:
On 21 Nov., 01:46, brzr...@gmail.com wrote:
dummy/
dummy_package/
__init__.py
moduleA.py
tests/
__init__.py
test.py
To avoid confusion, the directory that is the package should be named
as you want the imports to appear; e.g. if you want to 'import
foo.module_a', then name the directory containing 'module_a.py' as
'foo/'.

This often results in::

foo/
setup.py
foo/
__init__.py
module_a.py
test/
__init__.py
test_module_a.py

That is, the *project* directory (containing all the files) is named
'foo/'; the *package* directory (where all the implementation code is
found) is named 'foo/foo/', and the unit tests are found in the
directory 'foo/test/'.

That's normal, though if it confuses you you might want to rename the
project directory. I'm often doing development on multiple
version-control branches, so each project directory is named for the
branch it contains; within each of those, the same 'foo/' name is used
for the package directory.
I'm using Python 2.5.1. When I'm trying to call a function in
'moduleA' from 'test' it won't work unless I make the 'dummy'
folder a package as well. That's pretty weird. Does
'dummy_package' have to be in my pythonpath or something? How do I
reference moduleA from test?
You should install the package into a place where Python's import path
will find it. Read up on using the standard library 'distutils'
mechanism for this; it involves writing a 'setup.py' file to define
the parameters for installation of your package.
I would like to avoid making 'dummy' into a package as well.
Yes. The top-level directory is used for containing a number of files,
including 'setup.py', that should not be part of the installed
package.
Problem solved. I added 'dummy' to the PYTHONPATH. (Do I really have
to do that for every project I create?) Anyway, it works the way I'd
like it to now.
I hope the above makes it clearer what I prefer for this situation.

--
\ "True greatness is measured by how much freedom you give to |
`\ others, not by how much you can coerce others to do what you |
_o__) want." --Larry Wall |
Ben Finney
Nov 21 '07 #3

P: n/a
As a side note, I find much easier to drop a PTH file than messing
with pythonpath. If you are not familiar with PTH files, what I do is
this

1) Go to "C:\Program Files\Python25\Lib\site-packages" or whatever is
appropiate in your case.
2) Create a text file, name it something like "MyProjects.PTH" (note
the extension!)
3) in the file, just write the path of the folder that contains all
your projects (in my case, C:/docs/python)

The idea is to keep the main python installation separated from the
modules you are currently developing. Your python installation goes to
"program files/python" or "bin/python", and your personal projects go
somewhere else (usually inside your 'user' folder). This smooths many
things, like working with different versions of a package you are
developing.

On Nov 20, 2007 11:34 PM, Ben Finney <bi****************@benfinney.id.auwrote:
Jens <j3****@gmail.comwrites:
On 21 Nov., 04:16, Jens <j3n...@gmail.comwrote:
On 21 Nov., 01:46, brzr...@gmail.com wrote:
dummy/
dummy_package/
__init__.py
moduleA.py
tests/
__init__.py
test.py

To avoid confusion, the directory that is the package should be named
as you want the imports to appear; e.g. if you want to 'import
foo.module_a', then name the directory containing 'module_a.py' as
'foo/'.

This often results in::

foo/
setup.py
foo/
__init__.py
module_a.py
test/
__init__.py
test_module_a.py

That is, the *project* directory (containing all the files) is named
'foo/'; the *package* directory (where all the implementation code is
found) is named 'foo/foo/', and the unit tests are found in the
directory 'foo/test/'.

That's normal, though if it confuses you you might want to rename the
project directory. I'm often doing development on multiple
version-control branches, so each project directory is named for the
branch it contains; within each of those, the same 'foo/' name is used
for the package directory.
I'm using Python 2.5.1. When I'm trying to call a function in
'moduleA' from 'test' it won't work unless I make the 'dummy'
folder a package as well. That's pretty weird. Does
'dummy_package' have to be in my pythonpath or something? How do I
reference moduleA from test?

You should install the package into a place where Python's import path
will find it. Read up on using the standard library 'distutils'
mechanism for this; it involves writing a 'setup.py' file to define
the parameters for installation of your package.
I would like to avoid making 'dummy' into a package as well.

Yes. The top-level directory is used for containing a number of files,
including 'setup.py', that should not be part of the installed
package.
Problem solved. I added 'dummy' to the PYTHONPATH. (Do I really have
to do that for every project I create?) Anyway, it works the way I'd
like it to now.

I hope the above makes it clearer what I prefer for this situation.

--
\ "True greatness is measured by how much freedom you give to |
`\ others, not by how much you can coerce others to do what you |
_o__) want." --Larry Wall |
Ben Finney

--
http://mail.python.org/mailman/listinfo/python-list
Nov 21 '07 #4

P: n/a
On 21 Nov., 05:34, Ben Finney <bignose+hates-s...@benfinney.id.au>
wrote:
>
I hope the above makes it clearer what I prefer for this situation.
It does. You've been most helpful - thanks a lot! I'll bookmark this
thread and keep it under my pillow :-)
Nov 21 '07 #5

P: n/a
On Nov 21, 7:05 am, "Sergio Correia" <sergio.corr...@gmail.comwrote:
As a side note, I find much easier to drop a PTH file than messing
with pythonpath. If you are not familiar with PTH files, what I do is
this

1) Go to "C:\Program Files\Python25\Lib\site-packages" or whatever is
appropiate in your case.
2) Create a text file, name it something like "MyProjects.PTH" (note
the extension!)
3) in the file, just write the path of the folder that contains all
your projects (in my case, C:/docs/python)

The idea is to keep the main python installation separated from the
modules you are currently developing. Your python installation goes to
"program files/python" or "bin/python", and your personal projects go
somewhere else (usually inside your 'user' folder). This smooths many
things, like working with different versions of a package you are
developing.
Hi,

If I understand you correctly, you have the following directory
organisation
[Python installation]/site-packages/[MyProject.pth pointing to /home/
user/python-dev/]

And then you do your development in python-dev. But how do you manage
multiple development branches of the same program ?

My directory structure looks like this:
python-dev:
+ prog-branch1/
| + foo
| | + foo.py
| | + tests
| | + test_foo.py
| + bar
| + bar.py
| + tests
| + test_bar.py
+ prog-branch2/
+ foo
| + foo.py
| + tests
| + test_foo.py
+ bar
+ bar.py
+ tests
+ test_bar.py

bar/bar.py needs to import symbols from foo/foo.py . And bar/tests/
test_bar.py needs some symbols from both bar/bar.py and foo/foo.py

I don't understand how having python-dev in the .pth file solves the
problem.

In my case, I make all my imports relative to the root of my project:
bar/bar.py:
from foo.foo import some_foo

bar/tests/test_bar.py
from foo.foo import some_foo
from foo.bar import some_bar

The way I managed to make it work is by extending sys.path but I would
be happy to find a better solution:

bar/bar.py is actually:
import os, sys
sys.path.append( '..')

from foo.foo import some_foo

and bar/tests/test_bar.py is actually:
import os, sys
sys.path.append( os.path.join('..','..') )

from foo.foo import some_foo
from foo.foo import some_bar

What is not nice with this method is that every runnable test script
must extend unconditionally the sys.path . That creates some clutter.

Philippe
Nov 24 '07 #6

P: n/a
On 2007-11-24, BlueBird <ph**@freehackers.orgwrote:
On Nov 21, 7:05 am, "Sergio Correia" <sergio.corr...@gmail.comwrote:
And then you do your development in python-dev. But how do you manage
multiple development branches of the same program ?
If you are using SVN, you may want to check out 'combinator' by Divmod
(divmod.org). That tool manages SVN branches, and also switches Python when you
switch branches. Very handy.

Albert
Nov 26 '07 #7

P: n/a
Bluebird:

If you are using python 2.5, relative imports are no longer an issue:
http://docs.python.org/whatsnew/pep-328.html

That problem solved, what you sometimes want is to change the version
of your package. I just change the text in the PTH file, to point to
another version, and voilá (no need to move files around or to mess
with the code in the package itself). Probably you can write a script
that changes the PTH file from python itself.

Albert:

Thanks, that looks useful.
Nov 26 '07 #8

P: n/a
On Nov 20, 4:09 pm, Jens <j3n...@gmail.comwrote:
>
1) Should I put my unittests in a subdirectory? Does the subdirectory
have to be a package?
As others have suggested, this is a good way to organise your tests.
To avoid problems with the import path, look at nosetests [1]. This
allows you to run::

nosetests
main_package_name.subpackage1.test.test1:TestSomeC lass.test_func

to run the test_func function in the TestSomeClass test case.

Or alternatively::

nosetests main_package_name.subpackage1

to run all tests contained in the main_package_name.subpackage1
module.

This allows you to remove all of your 'if __name__ ==
"__main__":unittest.main()' code, as well as gets rid of the problem
with relative imports (which others solved using .pth files).

The only downside for me is that running nosetests takes at least half
a second (eg: import nose), compared to making the tests runnable
which can be _very_ quick. For test driven development it's nice to
have tests really really quick.

One gotcha; if you're converting from a 'if __name__ ==
"__main__":...' system; nosetests by default ignores files that are
executable (so you'll need to chmod -x main_package_name/subpackage1/
test/test1.py).
[1] nose: a discovery-based unittest extension - (http://
somethingaboutorange.com/mrl/projects/nose/).

Nov 28 '07 #9

P: n/a
On Nov 26, 5:07 pm, "Sergio Correia" <sergio.corr...@gmail.comwrote:
Bluebird:

If you are using python 2.5, relative imports are no longer an
issue:http://docs.python.org/whatsnew/pep-328.html
It does not solve my problem, or I missed something:

User@Phil_vaio /cygdrive/d/work/work/vy-dev/foo $ find . -name '*.py'
../bar/bar.py
../bar/tests/test_bar.py
../bar/tests/test_bar1.py
../bar/tests/__init__.py
../bar/__init__.py
../foo/foo.py
../foo/tests/__init__.py
../foo/__init__.py
../__init__.py

d:\work\work\vy-dev\foo>C:\Python25\python.exe bar/tests/test_bar.py
d:\work\work\vy-dev\foo
['d:\\work\\work\\vy-dev\\foo\\bar\\tests', 'C:\\Python25\\lib\\site-
packages\\h
gsvn-0.1.4-py2.5.egg', 'C:\\WINDOWS\\system32\\python25.zip', 'C:\
\Python25\\DLL
s', 'C:\\Python25\\lib', 'C:\\Python25\\lib\\plat-win', 'C:\\Python25\
\lib\\lib-
tk', 'C:\\Python25', 'C:\\Python25\\lib\\site-packages', 'C:\\Python25\
\lib\\sit
e-packages\\win32', 'C:\\Python25\\lib\\site-packages\\win32\\lib', 'C:
\\Python2
5\\lib\\site-packages\\Pythonwin', 'C:\\Python25\\lib\\site-packages\
\wx-2.8-msw
-ansi']
Traceback (most recent call last):
File "bar/tests/test_bar.py", line 8, in <module>
from ..bar import bar_f
ValueError: Attempted relative import in non-package

Dec 2 '07 #10

P: n/a
On Dec 2, 4:27 pm, BlueBird <p...@freehackers.orgwrote:
On Nov 26, 5:07 pm, "Sergio Correia" <sergio.corr...@gmail.comwrote:
Bluebird:
If you are using python 2.5, relative imports are no longer an
issue:http://docs.python.org/whatsnew/pep-328.html

It does not solve my problem, or I missed something:
Any idea what could be wrong in my code (in the previous message). Or
did I misunderstand relative import seriously ?

Philippe
Dec 7 '07 #11

P: n/a
BlueBird <ph**@freehackers.orgwrote:
On Dec 2, 4:27 pm, BlueBird <p...@freehackers.orgwrote:
>On Nov 26, 5:07 pm, "Sergio Correia" <sergio.corr...@gmail.comwrote:
>Bluebird:
If you are using python 2.5, relative imports are no longer an
issue:http://docs.python.org/whatsnew/pep-328.html

It does not solve my problem, or I missed something:

Any idea what could be wrong in my code (in the previous message). Or
did I misunderstand relative import seriously ?
Does this give you an idea?

0:tolot:/tmpmkdir t1
0:tolot:/tmptouch t1/__init__.py
0:tolot:/tmpecho print __name__ >t1/test.py
0:tolot:/tmppython t1/test.py
__main__
0:tolot:/tmppython -c "import t1.test"
t1.test

Marc
Dec 7 '07 #12

This discussion thread is closed

Replies have been disabled for this discussion.