python project layout

[Huron]

Hi,

What do you guys recommend in terms of python project layout, especially
unit tests layout ?
Zope has unit tests per packages, twisted has a big tests directory full of
tests ... and the file naming convention are also pretty differents ...
I guess there is part of "personnal choices" in those matters ... but there
is also good practice advices that can be gathered I think ...

- h

[Peter Hansen]

Huron wrote:

What do you guys recommend in terms of python project layout, especially
unit tests layout ?
Zope has unit tests per packages, twisted has a big tests directory full of
tests ... and the file naming convention are also pretty differents ...
I guess there is part of "personnal choices" in those matters ... but there
is also good practice advices that can be gathered I think ...

(This is the result of a few years of evolution of our source tree
conventions.) We place all tests in subfolders (named "tests") of the
folders containing the files under test. Unit tests are named for the
unit (module) they are testing, so serial.py would have a unit test file
at tests/serial_unit.py.

"Acceptance" tests (higher level usually black-box functional tests) are
in the "tests" folder of the top level source folder, along with any
unit tests that reside there, and take their name from the number of the
requirement that defined them. Since we're an XP team, we call these
requirements "stories", so the files are named exciting things like
"story001.py" and "story197.py". The specific numbers match up with
either index cards with descriptions of the requirements, or an online
"issue tracker", depending on the project.

We have a little (quite simple) internal utility that lets us easily run
all unit tests under the current folder, or all acceptance tests in the
project, with one command. (Specifically, "uix" and "stix" for unit and
story tests, respectively.) This "test runner" utility scans subfolders
(doing an os.walk basically) and executes each test script in a new
process, to avoid most potential undesirable interactions between tests.

The benefits are straightforward. Tests are easy to find for any file,
but don't "clutter up" the same folder as the files under test. The
naming conventions makes it easy to automate things like the test runner
or other utilities. Running tests (always from a console) takes only a
few seconds so we can do it often. Multiple projects are easily
supported in other directory trees.

-Peter

[Huron]

Hi Peter,

Thanks for you detailed reply.
The layout that you suggest sounds wise to me (I'm about to start a
project).
I confess that, coming from the java world, I've trouble to move away from
the paradigm "one class" = "one file" ... and to gather things inside
"python modules" ...

-- huron

[Huron]

Hi again,

For instance, if you have several classes implementing the same interface
(say, doing the same things with different strategies), how would you
organize that in terms of files (modules) and directories (packages) ?

-- huron

[Peter Hansen]

Huron wrote:

Hi Peter,
Thanks for you detailed reply.
The layout that you suggest sounds wise to me (I'm about to start a
project).

I wouldn't necessarily recommend something so complex (not that it's
particular complex, but it's more than just "flat") for a newcomer,
however. One of the few small issues you have to deal with in order to
get such a layout working properly is the "python path" issue, and how
to import modules that are in another folder.

Normally (aside from the standard library and extensions) only the
current directory is in sys.path. Any subfolders that contain
__init__.py modules are "packages" and you can import from them using
the "dotted" notation (e.g. "import mypkg.mymodule" will try to load a
file from ./mypkg/mymodule.py if there is also a ./mypkg/__init__.py
(even if that file is empty!)).

To import a module in the *parent* directory, however, is another story
entirely. You need to get it added to the sys.path, and that's one of
the things done (dynamically) by our test utilities, so we can be in the
tests subfolder and type "story015.py" and have it load modules in the
parent folder.

So in short, start simple and let your layout evolve as you need it to.
Don't try to start with a more complex layout than you need or know
how to handle. And if you're using a revision control system like
Subversion (and you better be using something! :-) ) then it's an easy
matter to rename or move folders at a later time, without losing your
revision history.

-Peter

[Peter Hansen]

Huron wrote:

For instance, if you have several classes implementing the same interface
(say, doing the same things with different strategies), how would you
organize that in terms of files (modules) and directories (packages) ?

A concrete example would probably help. What are you envisioning?

[Huron]

> A concrete example would probably help. What are you envisioning?

Let say you have an interface (zope.interface) IMyProduct and two classes
implementing the interface MyProduct1 and MyProduct2 ...
But I think I found the answer. One file each.
Python is just more flexible than java in the sens that you dont NEED to do
"one class" = "one file"

-- h