473,854 Members | 1,765 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Project organization and import

I'm using Python for what is becoming a sizeable project and I'm
already running into problems organizing code and importing packages.
I feel like the Python package system, in particular the isomorphism
between filesystem and namespace, doesn't seem very well suited for
big projects. However, I might not really understand the Pythonic way.
I'm not sure if I have a specific question here, just a general plea
for advice.

1) Namespace. Python wants my namespace heirarchy to match my
filesystem heirarchy. I find that a well organized filesystem
heirarchy for a nontrivial project will be totally unwieldy as a
namespace. I'm either forced to use long namespace prefixes, or I'm
forced to use "from foo import *" and __all__, which has its own set
of problems.

1a) Module/class collision. I like to use the primary class in a file
as the name of the file. However this can lead to namespace collisions
between the module name and the class name. Also it means that I'm
going to be stuck with the odious and wasteful syntax foo.foo
everywhere, or forced to use "from foo import *".

1b) The Pythonic way seems to be to put more stuff in one file, but I
believe this is categorically the wrong thing to do in large projects.
The moment you have more than one developer along with a revision
control system, you're going to want files to contain the smallest
practical functional blocks. I feel pretty confident saying that "put
more stuff in one file" is the wrong answer, even if it is the
Pythonic answer.

2) Importing and reloading. I want to be able to reload changes
without exiting the interpreter. This pretty much excludes "from foo
import *", unless you resort to this sort of hack:

http://www.python.org/search/hyperma...1993/0448.html

Has anyone found a systematic way to solve the problem of reloading in
an interactive interpreter when using "from foo import *"?
I appreciate any advice I can get from the community.

Martin

Mar 5 '07
49 3961
On 5 Mar 2007 23:35:00 -0800, Martin Unsal <ma*********@gm ail.comwrote:
On Mar 5, 11:06 am, "Chris Mellon" <arka...@gmail. comwrote:
I never advocated big files with many functional units - just files
that are "just big enough".

Then we're in total agreement. I'm not sure why you thought my
opinions were the result of baggage from other languages when you
don't seem to actually disagree with me.
Because you're advocating single class per file. A scan through the
standard library may be instructive, where there are some modules that
expose a single class (StringIO, pprint) and others that expose many,
and some that expose none at all. "smallest unit that it makes sense
to work on" and "single class" are totally different things. In any
case, as I hinted at, I prefer an organic, developer driven approach
to deciding these things, not handed down from above style guidelines.
You know your modules are broken up enough when you no longer have
conflicts.
Fewer dependencies between compilation units means a
faster rebuild-test turnaround.

I know all about incremental builds and I just don't think people use
small compilation units in C++ to make their builds faster. It
certainly never been the reason why I subdivided a source file.
Faster compile/debug/edit cycle is the main justification I've heard
for single class per file. The others are variations of your RCS
argument, which I don't think is justifiable for the above reasons. It
smells of the kind of "my developers are stupid" short sighted
management that kills projects.
Sure, but whats your goal here? If you're just testing something as
you work, then this works fine. If you're testing large changes, that
affect many modules, then you *need* to reload your world, because you
want to make sure that what you're testing is clean.

I don't think reload works for anything but trivial scripts. The
moment you use "from foo import bar" reload is broken.
The semantics of exactly what reload should do are tricky. Pythons
reload works in a sensible but limited way.

I agree that there is some subtlety there, and I appreciate your
example. However the fact that Python's module system essentially
forces you to use "from foo import *" and that reload is almost
entirely imcompatible with "from foo import *", I would say that
reload is essentially useless.
I'm still not sure why you believe this, since several counterexamples
where given. As an intellectual exercise, though, lets assume that
reload is totally broken and you just can't use it. Pretend it will
reformat your machine if you ever call it. Can you really think of no
other reason to use Python? You still haven't given any justification
for why a magic reload is essential to Python development when a) all
existing python development works fine without it and b) all existing
development in every other language works fine without it.
That said, nothing prevents you from using "from foo import Foo" if
Foo is all you need (or need most - you can combine this with import
foo).

Well "from foo import Foo" is just a special case of "from foo import
*". :) It still breaks reload. It still means you're restarting your
interpreter even to do the most trivial development cycle.
You're totally fixated on reload. I don't understand this. I'm totally
positive that your traditional development experience has not been in
an environment where you could effortlessly slot in new code to a
running image. Why do you demand it from Python?

Also, the difference between "from foo import Bar" and "from foo
import *" is that the former is limited in scope (you're adding a
limited set of explicit names to your namespace) and is futureproof
(additional names exported from foo won't clash with vars in the
importing module with unknown effects). The reason why one is common
and accepted and the other is frowned upon has nothing to do with
reload().
I wonder what environments you worked in before that actually had a
reliable and gotcha free version of reload?

I'm perfectly well aware that I'm not going to be able to reload a
widget in the middle of a running GUI app, for example. I'm not
looking for gotcha free, I'll settle for minimally useful.
Then reload() as is is what you want.
Here's an analogy. In C, you can do an incremental build and run your
modified application without having to first reboot your computer. In
Python, where reload() is essentially the incremental build process,
and the interpreter is essentially a virtual machine, you guys are
saying that my best option is to just "reboot" the virtual machine to
make sure I have a "clean slate". It may be the path of least
resistance, but to say that it is necessary or inevitable is 1960s
mainframe thinking.
But you do need to restart the application image. The python
interpreter is not an emulator. You're drawing incompatible analogies
and making unjustified assumptions based on them. reload() is not an
incremental build process, and starting a new Python instance is not
rebooting your machine. This is just not a justifiable comparison.
Mar 6 '07 #21
Martin Unsal <ma*********@gm ail.comwrote:
On Mar 5, 10:06 pm, a...@mac.com (Alex Martelli) wrote:
My favorite way of working: add a test (or a limited set of tests) for
the new or changed feature, run it, check that it fails, change the
code, rerun the test, check that the test now runs, rerun all tests to
see that nothing broke, add and run more tests to make sure the new code
is excellently covered, rinse, repeat. Occasionally, to ensure the code
stays clean, stop to refactor, rerunning tests as I go.

From the way you describe your workflow, it sounds like you spend very
little time working interactively in the interpreter. Is that the case
or have I misunderstood?
I often do have an interpreter open in its own window, to help me find
out something or other, but you're correct that it isn't where I "work";
I want all tests to be automated and repeatable, after all, so they're
better written as their own scripts and run in the test-framework. I
used to use a lot of doctests (often produced by copy and paste from an
interactive interpreter session), but these days I lean more and more
towards unittest and derivatives thereof.

Sometimes, when I don't immediately understand why a test is failing
(or, at times, why it's unexpectedly succeeding _before_ I have
implemented the feature it's supposed to test!-), I stick a
pdb.set_trace() call at the right spot to "look around" (and find out
how to fix the test and/or the code) -- I used to use "print" a lot for
such exploration, but the interactive interpreter started by pdb is
often handier (I can look at as many pieces of data as I need to find
out about the problem). I still prefer to run the test[s] within the
test framework, getting interactive only at the point where I want to
be, rather than running the tests from within pdb to "set breakpoints"
manually -- not a big deal either way, I guess.
Alex
Mar 6 '07 #22
Jorge Godoy <jg****@gmail.c omwrote:
...
My favorite way of working: add a test (or a limited set of tests) for
the new or changed feature, run it, check that it fails, change the
code, rerun the test, check that the test now runs, rerun all tests to
see that nothing broke, add and run more tests to make sure the new code
is excellently covered, rinse, repeat. Occasionally, to ensure the code
stays clean, stop to refactor, rerunning tests as I go.

I believe this is a distinct case. When we write tests we're worried with the
system itself.
Not sure I get what you mean; when I write tests, just as when I write
production code, I'm focused (not worried:-) about the application
functionality I'm supposed to deliver. The language mostly "gets out of
my way" -- that's why I like Python, after all:-).

When using the interactive interpreter we're worried with how
to best use the language. There might be some feature of the system related
to that investigation, but there might be not. For example: "what are the
methods provided by this object?" or "which approach is faster for this loop?"
I do generally keep an interactive interpreter running in its own
window, and help and dir are probably the functions I call most often
there. If I need to microbenchmark for speed, I use timeit (which I
find far handier to use from the commandline). I wouldn't frame this as
"worried with how to best use the language" though; it's more akin to a
handy reference manual (I also keep a copy of the Nutshell handy for
exactly the same reason -- some things are best looked up on paper).

I won't write a test case to test loop speed. But I'd poke with the
interpreter and if the environment gets a bit big to setup then I'd go to the
text editor as I said.
I don't really see "getting a bit big to setup" as the motivation for
writing automated, repeatable tests (including load-tests, if speed is
such a hot topic in your case); rather, the key issue is, will you ever
want to run this again? For example, say you want to check the relative
speeds of approaches A and B -- if you do that in a way that's not
automated and repeatable (i.e., not by writing scripts), then you'll
have to repeat those manual operations exactly every time you refactor
your code, upgrade Python or your OS or some library, switch to another
system (HW or SW), etc, etc. Even if it's only three or four steps, who
needs the aggravation? Almost anything worth doing (in the realm of
testing, measuring and variously characterizing software, at least) is
worth automating, to avoid any need for repeated manual labor; that's
how you get real productivity, by doing ever less work yourself and
pushing ever more work down to your computer.
Alex
Mar 6 '07 #23

Bruno Desthuilliers wrote:
<imho>
Which is not a problem. reload() is of very limited use for any
non-trivial stuff.
</imho>
Now that I've heard this from 5 different people it might be sinking
in. :) :) I really do appreciate all of you taking the time to explain
this to me.

When I started using Python a few years ago I was very excited about
the fact that it was an interpreted language and offered a more
interactive workflow than the old compile-link-test workflow. As my
project has grown to be pretty sizeable by Python standards, I tried
to continue taking advantage of the tight, reload-based, interpreted-
language workflow and it's become really cumbersome, which is
disappointing. However y'all are right, giving up on reload() doesn't
mean Python is inadequate for large projects, just that it doesn't
live up entirely to what I perceived as its initial promise. Once I
adjust my mindset and workflow for a life without reload(), I'll
probably be better off.

I'd like to point out something though. More than one of the people
who responded have implied that I am bringing my prior-language
mindset to Python, even suggesting that my brain isn't built for
Python. ;) In fact I think it's the other way around. I am struggling
to take full advantage of the fact that Python is an interpreted
language, to use Python in the most "Pythonic" way. You guys are
telling me that's broken and I should go back to a workflow that is
identical in spirit, and not necessarily any faster than I would use
with a compiled language. While that might be the right answer in
practice, I don't feel like it's a particularly "good" answer, and it
confirms my initial impression that Python package management is
broken.

I think you should be asking yourselves, "Did we all abandon reload()
because it is actually an inferior workflow, or just because it's
totally broken in Python?"

I have one question left but I'll ask that in a separate post.

Martin

Mar 6 '07 #24
On Mar 5, 2:18 pm, Bruno Desthuilliers
<bdesth.quelque ch...@free.quel quepart.frwrote :
Martin Unsal a écrit :
For example, say
you want to organize the widgets package as follows:
widgets/scrollbar/*.py
widgets/form/*.py
widgets/common/util.py
Other than messing around with PYTHONPATH, which is horrible, I don't
see how to import util.py from the widget code.

Some of us still manage to do so without messing with PYTHONPATH.
How exactly do you manage it?

The only way I can see to do it is to have widgets/__init__.py look
something like this:

from common import util
from scrollbar import Scrollbar
from form import Form

Then Scrollbar.py doesn't have to worry about importing util, it just
assumes that util is already present in its namespace.

BUT ... this means that Scrollbar.py can only be loaded in the
interpreter as part of package "widgets". You can't run an interpreter
and type "import widgets.scrollb ar.Scrollbar" and start going to town,
because Scrollbar doesn't import its own dependencies.

So what I want to clarify here: Do Python programmers try to design
packages so that each file in the package can be individually loaded
into the interpreter and will automatically import its own
dependencies? Or do you design packages so they can only be used by
importing from the top level and running the top level __init__.py?

I hope that made sense. :)

Martin

Mar 6 '07 #25
I'd like to point out something though. More than one of the people
who responded have implied that I am bringing my prior-language
mindset to Python, even suggesting that my brain isn't built for
Python. ;) In fact I think it's the other way around. I am struggling
to take full advantage of the fact that Python is an interpreted
language, to use Python in the most "Pythonic" way. You guys are
telling me that's broken and I should go back to a workflow that is
identical in spirit, and not necessarily any faster than I would use
with a compiled language. While that might be the right answer in
practice, I don't feel like it's a particularly "good" answer, and it
confirms my initial impression that Python package management is
broken.

I think you should be asking yourselves, "Did we all abandon reload()
because it is actually an inferior workflow, or just because it's
totally broken in Python?"
Sorry, but I fail to see the point of your argumentation.

Reloading a module means that you obviously have some editor open you code
your module in, and an interactive interpreter running where you somehow
have to make the

reload(module)

line (re-)appear, and then most probably (unless the pure reloading itself
triggers some testing code) some other line that e.g. instantiates a class
defined in "module"

Now how exactly does that differ from having a test.py file containing

import module
<do-something>

and a commandline sitting there with a

python test.py

waiting to be executed, easily brought back by a single key-stroke.

Especially if <do-somethingbecome s more that some easy lines brought back
by the command line history.

I've been writing python for a few years now, to programs the size of a few
K-lines, and _never_ felt the slightest need to reload anything. And as
there have been quite a few discussions like this in the past few years,
IMHO reload is a wart and should be removed.

Diez
Mar 6 '07 #26
On 6 Mar 2007 08:42:00 -0800, Martin Unsal <ma*********@gm ail.comwrote:
On Mar 5, 2:18 pm, Bruno Desthuilliers
<bdesth.quelque ch...@free.quel quepart.frwrote :
Martin Unsal a écrit :
For example, say
you want to organize the widgets package as follows:
widgets/scrollbar/*.py
widgets/form/*.py
widgets/common/util.py
Other than messing around with PYTHONPATH, which is horrible, I don't
see how to import util.py from the widget code.
Some of us still manage to do so without messing with PYTHONPATH.

How exactly do you manage it?

The only way I can see to do it is to have widgets/__init__.py look
something like this:

from common import util
from scrollbar import Scrollbar
from form import Form

Then Scrollbar.py doesn't have to worry about importing util, it just
assumes that util is already present in its namespace.

BUT ... this means that Scrollbar.py can only be loaded in the
interpreter as part of package "widgets". You can't run an interpreter
and type "import widgets.scrollb ar.Scrollbar" and start going to town,
because Scrollbar doesn't import its own dependencies.

So what I want to clarify here: Do Python programmers try to design
packages so that each file in the package can be individually loaded
into the interpreter and will automatically import its own
dependencies? Or do you design packages so they can only be used by
importing from the top level and running the top level __init__.py?

I hope that made sense. :)
Scrollbar *can't* assume that util will be present in its namespace,
because it won't be unless it imports it. Scrollbar needs to import
its own dependencies. But why do you think thats a problem?
Martin

--
http://mail.python.org/mailman/listinfo/python-list
Mar 6 '07 #27
On Mar 6, 6:07 am, "Chris Mellon" <arka...@gmail. comwrote:
Because you're advocating single class per file.
What I actually said was "Smallest practical functional block." I
never said one class per file, in fact I generally have more than one
class per file. Nonetheless I frequently have a class which has the
same name as the file it's contained in, which is where I start having
trouble.
What you said was A scan through the
standard library may be instructive, where there are some modules that
expose a single class (StringIO, pprint) and others that expose many,
and some that expose none at all.
AHA! Here we see the insidious Python package system at work! ;)

I said "file" and you assume that I am talking about the exposed
namespace. Files should not have to be isomorphic with namespace! A
package that exposes many classes may still use one class per file if
it wants to.
In any
case, as I hinted at, I prefer an organic, developer driven approach
to deciding these things, not handed down from above style guidelines.
PRECISELY. And in the case of Python, package stucture is dictated,
not by a style guideline, but by the design flaws of Python's package
system.

Martin

Mar 6 '07 #28
On 6 Mar 2007 09:09:13 -0800, Martin Unsal <ma*********@gm ail.comwrote:
On Mar 6, 6:07 am, "Chris Mellon" <arka...@gmail. comwrote:
Because you're advocating single class per file.

What I actually said was "Smallest practical functional block." I
never said one class per file, in fact I generally have more than one
class per file. Nonetheless I frequently have a class which has the
same name as the file it's contained in, which is where I start having
trouble.
You do? Or do you only have trouble because you don't like using "from
foo import Foo" because you need to do more work to reload such an
import?
>
What you said was A scan through the
standard library may be instructive, where there are some modules that
expose a single class (StringIO, pprint) and others that expose many,
and some that expose none at all.

AHA! Here we see the insidious Python package system at work! ;)

I said "file" and you assume that I am talking about the exposed
namespace. Files should not have to be isomorphic with namespace! A
package that exposes many classes may still use one class per file if
it wants to.
What makes you think that the exposed namespace has to be isomorphic
with the filesystem? Further, why do you think doing so is bad? People
do it because it's convenient and simple, not because its necessary.
Why don't you like filesystems?
In any
case, as I hinted at, I prefer an organic, developer driven approach
to deciding these things, not handed down from above style guidelines.

PRECISELY. And in the case of Python, package stucture is dictated,
not by a style guideline, but by the design flaws of Python's package
system.
What design flaws are those? Is it because you're trying to have
packages as part of your project without installing them on your
PYTHONPATH somewhere?

If you want to break a module internally into multiple files, then
make it a package. To an importer, they're almost indistinguishab le.
If you want to break a module into multiple packages and then stick
the files that make up the package in bizarre spots all over the
filesystem, can you give a reason why?
Mar 6 '07 #29
On Mar 6, 8:56 am, "Chris Mellon" <arka...@gmail. comwrote:
Scrollbar *can't* assume that util will be present in its namespace,
because it won't be unless it imports it. Scrollbar needs to import
its own dependencies. But why do you think thats a problem?
OK, maybe I'm totally missing something here, but you can't do
"import ../util/common" in Python can you?

Look at the directory structure in my original post. How does
Scrollbar.py import its dependencies from common.py, without relying
on PYTHONPATH?

Martin

Mar 6 '07 #30

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
10932
by: Steven T. Hatton | last post by:
I think Danny was one cup of coffee shy of full consciousness when he wrote this, but the gist of it makes sens to me: "C++ Project Organization Guidelines Last updated May 26, 2005. http://www.informit.com/guides/content.asp?g=cplusplus&seqNum=175 Last week's article about inline functions subtly brought into the limelight another important issue, namely how to organize the files of a typical C++ program, or project. This week I...
2
2169
by: Lasse Vågsæther Karlsen | last post by:
I am slowly learning Python and I'm already starting to write some minor modules for myself. Undoubtedly there are better modules available either built-in or 3rd party that do the same as mine and much more but I need to learn it one way or another anyway. What I'm wondering about is module organization. I created my own directory for storing my modules and added the full path to this to PYTHONPATH (Windows XP platform).
10
9933
by: TokiDoki | last post by:
Hello there, I have been programming python for a little while, now. But as I am beginning to do more complex stuff, I am running into small organization problems. It is possible that what I want to obtain is not possible, but I would like the advice of more experienced python programmers. I am writing a relatively complex program in python that has now around 40 files.
8
7866
by: Neil Robbins | last post by:
I have created a setup project using the setup wizard and am now editing the properties of this project. I want to replace the default banner with one of my own. Could someone tell me what the dimensions of the banner should be (in pixels preferably). I can do a rough match, but I'd rather things were a bit more perfect. As always any help would be greatly appreciated. Neil R
0
10672
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10740
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10362
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9510
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7909
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7076
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5937
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4550
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3182
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.