473,847 Members | 1,709 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Project organization and import

I'm using Python for what is becoming a sizeable project and I'm
already running into problems organizing code and importing packages.
I feel like the Python package system, in particular the isomorphism
between filesystem and namespace, doesn't seem very well suited for
big projects. However, I might not really understand the Pythonic way.
I'm not sure if I have a specific question here, just a general plea
for advice.

1) Namespace. Python wants my namespace heirarchy to match my
filesystem heirarchy. I find that a well organized filesystem
heirarchy for a nontrivial project will be totally unwieldy as a
namespace. I'm either forced to use long namespace prefixes, or I'm
forced to use "from foo import *" and __all__, which has its own set
of problems.

1a) Module/class collision. I like to use the primary class in a file
as the name of the file. However this can lead to namespace collisions
between the module name and the class name. Also it means that I'm
going to be stuck with the odious and wasteful syntax foo.foo
everywhere, or forced to use "from foo import *".

1b) The Pythonic way seems to be to put more stuff in one file, but I
believe this is categorically the wrong thing to do in large projects.
The moment you have more than one developer along with a revision
control system, you're going to want files to contain the smallest
practical functional blocks. I feel pretty confident saying that "put
more stuff in one file" is the wrong answer, even if it is the
Pythonic answer.

2) Importing and reloading. I want to be able to reload changes
without exiting the interpreter. This pretty much excludes "from foo
import *", unless you resort to this sort of hack:

http://www.python.org/search/hyperma...1993/0448.html

Has anyone found a systematic way to solve the problem of reloading in
an interactive interpreter when using "from foo import *"?
I appreciate any advice I can get from the community.

Martin

Mar 5 '07
49 3956
Martin Unsal a écrit :
On Mar 5, 9:15 am, "Chris Mellon" <arka...@gmail. comwrote:
(snip)
There are myriad other benefits of breaking up large files into
functional units. Integration history, refactoring, reuse, as I
mentioned. Better clarity of design. Easier communication and
coordination within a team. What's the down side? What's the advantage
of big files with many functional units?
What is a "big file" ?

(snip)
However when I talk about what I think is "wrong" with the Pythonic
way, obviously that's just my opinion formed by my own experience.
Your own experience *with Python* ? or any close-enough language ? Or
your experience with C++ ?

(snip)
Python still takes time to load & "precompile ".
compile. To byte-code, FWIW. Not "load & precompile". And - apart from
the top-level script - only modified modules get recompiled.
That time is becoming
significant for me even in a modest sized project;
On what hardware are you working ??? I have my interpreter up and
running in a couple milliseconds, and my box is a poor athlon xp1200/256.
I imagine it would
be pretty awful in a multimillion line project.
I still wait to see a multimillion line project in Python !-)

If you find yourself in this situation, then you there's certainly
something totally wrong in the way you (and/or your team) design and code.

But anyway - remember that only the modified modules get recompiled.
No matter how fast it is, I'd rather reload one module than exit my
interpreter and reload the entire world.
Did you actually *tried* it ?
This is not a problem for Python as scripting language. This is a real
problem for Python as world class application development language.
Sorry to have to say so, but this is total bullshit IMHO - which is
based on working experience.
>>Nobody I know uses reload() for anything more than trivial "as
you work" testing in the interpreter. It's not reliable or recommended
for anything other than that.


That too... although I think that's unfortunate. If reload() were
reliable, would you use it?
I wouldn't. It easier to rerun a simple test script and keep the
interpreter opened with full state - and you're sure you have the
correct desired state then.
>>This is
still a much faster way than compiling any but the most trivial of
C/C++ modules.

I'm with you there! I love Python and I'd never go back to C/C++. That
doesn't change my opinion that Python's import mechanism is an
impediment to developing large projects in the language.
What about basing your opinion on facts ? What about going with the
language instead of fighting against it ?
>
>>If you don't like working with explicit namespaces, you've probably
chosen the wrong language.


I never said that. I like foo.Bar(), I just don't like typing
foo.Foo() and bar.Bar(), which is a waste of space; syntax without
semantics.
May I say that the problem here comes from your insistance on putting
each class in a single module ?
>
>>I propose that the technique most amenable to source code management
is for a single file (or RCS level module, if you have a locking RCS)
to have everything that it makes sense to edit or change for a
specific feature.


Oh, I agree completely. I think we're using the exact same criterion.
I really doubt you do. What Chris is talking about is grouping together
what usually needs to change together.
A class is a self-contained feature with a well defined interface,
So is a function. Should we put any single function in a separate module
then ?
just what you'd want to put in it's own file. (Obviously there are
trivial classes which don't implement features, and they don't need
their own files.)

>>You're also placing far too much emphasis on reload. Focus yourself on
unit tests and environment scripts instead. These are more reliable
and easier to validate than reload() in a shell.


I think this is the crux of my frustration. I think reload() is
unreliable and hard to validate because Python's package management is
broken.
I think the "crux of your frustation" comes from your a priori. Fighting
against a language can only bring you into frustration. If the language
don't fit your brain - which is perfectly legitimate - then use another
one - but don't blame the language for it.
Mar 5 '07 #11
Bruno Desthuilliers <bd************ *****@free.quel quepart.frwrote :
>I don't reload... When my investigative tests gets bigger I write a script
and run it with the interpreter. It is easy since my text editor can call
Python on a buffer (I use Emacs).
That's interesting, is this workflow pretty universal in the Python
world?

I don't know, but that's also mostly how I do work.
My favorite way of working: add a test (or a limited set of tests) for
the new or changed feature, run it, check that it fails, change the
code, rerun the test, check that the test now runs, rerun all tests to
see that nothing broke, add and run more tests to make sure the new code
is excellently covered, rinse, repeat. Occasionally, to ensure the code
stays clean, stop to refactor, rerunning tests as I go.

I'm also keen on bigger tests (integration tests, as well as system
tests for regressions, acceptance, etc), but of course I don't run those
anywhere as frequently (they're not part of my daily workflow, iterated
multiple times per day -- more like a "nightly run" kind of thing, or
for special occasions such as just before committing into HEAD... I'm
somewhat of a stickler about HEAD *always* passing *all* tests...).

Not exactly TDD, please note -- I tend to start the cycle with a few
tests (not strictly just one), implement some large chunk of the
new/changed stuff, and add "coverage" and "boundary cases" tests towards
the end of the cycle (more often than not I don't need further changes
to satisfy the coverage and boundary-case tests, because of the "large
chunk" thing). So, a TDD purist would blast me for heresy.

Nevertheless, having tried everything from pure TDD to papertrail-heavy
waterfall (including the "toss the bits over the wall to QA", shudder!)
to typical Chaos Driven Development, in over a quarter century of
experience, this almost-TDD is what works best for me -- in Python, C,
and C++, at least (it's been a long time, if ever, since I did enough
production Java, Haskell, Ruby, SML, assembly, Perl, bash, Fortran,
Cobol, Objective C, Tcl, awk, Scheme, PL/I, Rexx, Forth, Pascal,
Modula-2, or Basic, to be sure that the same approach would work well in
each of these cases, though I have no reason to think otherwise).
Alex
Mar 6 '07 #12
On Mar 5, 11:06 am, "Chris Mellon" <arka...@gmail. comwrote:
I never advocated big files with many functional units - just files
that are "just big enough".
Then we're in total agreement. I'm not sure why you thought my
opinions were the result of baggage from other languages when you
don't seem to actually disagree with me.
Fewer dependencies between compilation units means a
faster rebuild-test turnaround.
I know all about incremental builds and I just don't think people use
small compilation units in C++ to make their builds faster. It
certainly never been the reason why I subdivided a source file.
Sure, but whats your goal here? If you're just testing something as
you work, then this works fine. If you're testing large changes, that
affect many modules, then you *need* to reload your world, because you
want to make sure that what you're testing is clean.
I don't think reload works for anything but trivial scripts. The
moment you use "from foo import bar" reload is broken.
The semantics of exactly what reload should do are tricky. Pythons
reload works in a sensible but limited way.
I agree that there is some subtlety there, and I appreciate your
example. However the fact that Python's module system essentially
forces you to use "from foo import *" and that reload is almost
entirely imcompatible with "from foo import *", I would say that
reload is essentially useless.
That said, nothing prevents you from using "from foo import Foo" if
Foo is all you need (or need most - you can combine this with import
foo).
Well "from foo import Foo" is just a special case of "from foo import
*". :) It still breaks reload. It still means you're restarting your
interpreter even to do the most trivial development cycle.
I wonder what environments you worked in before that actually had a
reliable and gotcha free version of reload?
I'm perfectly well aware that I'm not going to be able to reload a
widget in the middle of a running GUI app, for example. I'm not
looking for gotcha free, I'll settle for minimally useful.

Here's an analogy. In C, you can do an incremental build and run your
modified application without having to first reboot your computer. In
Python, where reload() is essentially the incremental build process,
and the interpreter is essentially a virtual machine, you guys are
saying that my best option is to just "reboot" the virtual machine to
make sure I have a "clean slate". It may be the path of least
resistance, but to say that it is necessary or inevitable is 1960s
mainframe thinking.

Martin

Mar 6 '07 #13
On Mar 5, 3:11 pm, Bruno Desthuilliers
<bdesth.quelque ch...@free.quel quepart.frwrote :
Your own experience *with Python* ?
No, my experience with Visual Basic. ;)

Of course my experience with Python!

Sorry, I can continue writing snarky replies to your snarky comments
but that won't get us anywhere productive. Instead I think the
following really gets to the crux of the issue.
May I say that the problem here comes from your insistance on putting
each class in a single module ?
No, it doesn't.

It really doesn't matter how many classes you have in a module; either
you use "from foo import bar", or you are stuck with a file structure
that is isomorphic to your design namespace.

The former breaks reload; the latter breaks large projects.

Martin

Mar 6 '07 #14
On Mar 5, 10:06 pm, a...@mac.com (Alex Martelli) wrote:
My favorite way of working: add a test (or a limited set of tests) for
the new or changed feature, run it, check that it fails, change the
code, rerun the test, check that the test now runs, rerun all tests to
see that nothing broke, add and run more tests to make sure the new code
is excellently covered, rinse, repeat. Occasionally, to ensure the code
stays clean, stop to refactor, rerunning tests as I go.
>From the way you describe your workflow, it sounds like you spend very
little time working interactively in the interpreter. Is that the case
or have I misunderstood?

Martin

Mar 6 '07 #15
En Tue, 06 Mar 2007 04:57:18 -0300, Martin Unsal <ma*********@gm ail.com>
escribió:
On Mar 5, 10:06 pm, a...@mac.com (Alex Martelli) wrote:
>My favorite way of working: add a test (or a limited set of tests) for
the new or changed feature, run it, check that it fails, change the
code, rerun the test, check that the test now runs, rerun all tests to
[...]
From the way you describe your workflow, it sounds like you spend very
little time working interactively in the interpreter. Is that the case
or have I misunderstood?
FWIW, I only work interactively with the interpreter just to test some
constructs, or use timeit, or check code posted here... Never to develop
production code. That's why I don't care at all about reload(), by example.

--
Gabriel Genellina

Mar 6 '07 #16
In article <11************ **********@v33g 2000cwv.googleg roups.com>,
"Martin Unsal" <ma*********@gm ail.comwrote:
That too... although I think that's unfortunate. If reload() were
reliable, would you use it? Do you think it's inherently unreliable,
that is, it couldn't be fixed without fundamentally breaking the
Python language core?
I wrote a module that wraps __import__ and tracks the dependencies of
imports. It then allows you to unload any modules whose source have
changed. That seemed to work out nicely for multi-module projects.

However, one problem I ran into was that dynamic libraries don't get
reloaded, so if you are doing hybrid C++/Python development then this
doesn't help - you still have to restart the whole python process to
pick up changes in your C++ code.

I also didn't do a ton of testing. It worked for a few small projects
I was working on, but I stopped using it once I ran into the dynamic
library thing, and at this point I'm used to just restarting python
each time. I'm sure there are some odd things that some python modules
could do that would interfere with the automatic reloading code I
wrote.

If you're interested in the code, drop me an email.

Dave
Mar 6 '07 #17
Martin Unsal a écrit :
On Mar 5, 3:11 pm, Bruno Desthuilliers
<bdesth.quelque ch...@free.quel quepart.frwrote :
>>Your own experience *with Python* ?


No, my experience with Visual Basic. ;)

Of course my experience with Python!
Sorry but this was really not obvious.
Sorry, I can continue writing snarky replies to your snarky comments
but that won't get us anywhere productive.
You're right - sorry.
Instead I think the
following really gets to the crux of the issue.
>>May I say that the problem here comes from your insistance on putting
each class in a single module ?

No, it doesn't.

It really doesn't matter how many classes you have in a module; either
you use "from foo import bar", or you are stuck with a file structure
that is isomorphic to your design namespace.

The former breaks reload;
<imho>
Which is not a problem. reload() is of very limited use for any
non-trivial stuff.
</imho>

Mar 6 '07 #18
al***@mac.com (Alex Martelli) writes:
Bruno Desthuilliers <bd************ *****@free.quel quepart.frwrote :
>>I don't reload... When my investigative tests gets bigger I write a script
and run it with the interpreter. It is easy since my text editor can call
Python on a buffer (I use Emacs).

That's interesting, is this workflow pretty universal in the Python
world?

I don't know, but that's also mostly how I do work.

My favorite way of working: add a test (or a limited set of tests) for
the new or changed feature, run it, check that it fails, change the
code, rerun the test, check that the test now runs, rerun all tests to
see that nothing broke, add and run more tests to make sure the new code
is excellently covered, rinse, repeat. Occasionally, to ensure the code
stays clean, stop to refactor, rerunning tests as I go.
I believe this is a distinct case. When we write tests we're worried with the
system itself. When using the interactive interpreter we're worried with how
to best use the language. There might be some feature of the system related
to that investigation, but there might be not. For example: "what are the
methods provided by this object?" or "which approach is faster for this loop?"

I won't write a test case to test loop speed. But I'd poke with the
interpreter and if the environment gets a bit big to setup then I'd go to the
text editor as I said.
--
Jorge Godoy <jg****@gmail.c om>
Mar 6 '07 #19
"Martin Unsal" <ma*********@gm ail.comwrites:
On Mar 5, 11:06 am, "Chris Mellon" <arka...@gmail. comwrote:
>I never advocated big files with many functional units - just files
that are "just big enough".

Then we're in total agreement. I'm not sure why you thought my
opinions were the result of baggage from other languages when you
don't seem to actually disagree with me.
I believe the reason was that you were advocating one class per file. "big
enough" might be more classes. Or fewer... :-)
I agree that there is some subtlety there, and I appreciate your
example. However the fact that Python's module system essentially
forces you to use "from foo import *" and that reload is almost
entirely imcompatible with "from foo import *", I would say that
reload is essentially useless.
The don't force you to that... There are many modules that do, but they are
generally glueing your Python code to some other language (usually C) written
code. This is common for GUI development, for example.

In fact, it is rare to me -- mathematics, statistics, database, web
development, testing -- to use this construction. There are no modules that
demand that.

And you can also write:

from foo import Bar, Baz

or even

from foo import Bar as B1, Baz as B2 # OUCH! ;-)
Well "from foo import Foo" is just a special case of "from foo import
*". :) It still breaks reload. It still means you're restarting your
interpreter even to do the most trivial development cycle.
That's what you get when you're working with instances of Foo... I believe
that for classmethods this would work right. So, again, it depends on your
code, how it is structured (and how it can be structured), etc.
Here's an analogy. In C, you can do an incremental build and run your
modified application without having to first reboot your computer. In
Python, where reload() is essentially the incremental build process,
and the interpreter is essentially a virtual machine, you guys are
saying that my best option is to just "reboot" the virtual machine to
make sure I have a "clean slate". It may be the path of least
resistance, but to say that it is necessary or inevitable is 1960s
mainframe thinking.
How can you reload C code that would affect already running code --
ie. existing data, pointers, etc. -- without reloading the full program? Even
changing and reloading a dynamic library wouldn't do that to already existing
code, so you'd have to "reboot" your application as well.

--
Jorge Godoy <jg****@gmail.c om>
Mar 6 '07 #20

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
10932
by: Steven T. Hatton | last post by:
I think Danny was one cup of coffee shy of full consciousness when he wrote this, but the gist of it makes sens to me: "C++ Project Organization Guidelines Last updated May 26, 2005. http://www.informit.com/guides/content.asp?g=cplusplus&seqNum=175 Last week's article about inline functions subtly brought into the limelight another important issue, namely how to organize the files of a typical C++ program, or project. This week I...
2
2168
by: Lasse Vågsæther Karlsen | last post by:
I am slowly learning Python and I'm already starting to write some minor modules for myself. Undoubtedly there are better modules available either built-in or 3rd party that do the same as mine and much more but I need to learn it one way or another anyway. What I'm wondering about is module organization. I created my own directory for storing my modules and added the full path to this to PYTHONPATH (Windows XP platform).
10
9933
by: TokiDoki | last post by:
Hello there, I have been programming python for a little while, now. But as I am beginning to do more complex stuff, I am running into small organization problems. It is possible that what I want to obtain is not possible, but I would like the advice of more experienced python programmers. I am writing a relatively complex program in python that has now around 40 files.
8
7866
by: Neil Robbins | last post by:
I have created a setup project using the setup wizard and am now editing the properties of this project. I want to replace the default banner with one of my own. Could someone tell me what the dimensions of the banner should be (in pixels preferably). I can do a rough match, but I'd rather things were a bit more perfect. As always any help would be greatly appreciated. Neil R
0
10647
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10706
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
9482
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
7056
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5719
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5911
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4528
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4119
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
3164
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.