473,842 Members | 1,847 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

How to organize Python files in a (relatively) big project

Hello there,

I have been programming python for a little while, now. But as I am
beginning to do more complex stuff, I am running into small organization
problems.
It is possible that what I want to obtain is not possible, but I would
like the advice of more experienced python programmers.

I am writing a relatively complex program in python that has now around
40 files.
At first, I had all of my files in one single directory, but now, with
the increasing number of files, it is becoming hard to browse my directory.
So, I would want to be able to divide the files between 8 directory,
according to their purpose. The problem is that it breaks the 'import's
between my files. And besides,AFAIK, there is no easy way to import a
file that is not in a subdirectory of the current file (I suppose I
could adjust the os.path in every file, but that seems not very elegant
to me).
I thought about turning the whole into a package. But I have to change
every 'import module_name' into
'from package_name.su b_directory_nam e import module_name'
which is a little bit time consuming and not very flexible (if I change
my mind about the subdirectory a file belongs to, I will have to track
again every import to correct them)

So, basically, here is the point: is there an 'elegant' way to keep my
files in separate directory and still be able to import between them
with simple 'import filename'?

And if not, what would be the standard way to organize numerous python
files belonging to the same project?

Thank you,

TokiDoki
Oct 19 '05 #1
10 9931
I have this problem myself, and as I am a recent Python convert my
aproach may not be conventional - but as it is working for me I thought
I would share.

First off you need to split your code into logical objects. In my case
I have an obvious client and server object, But within each I have
individual stand alone major objects and so my directory structure
reflects this...

Prog Dir
-- MainApp.py
Bin
__init__.py
--- main.py
--- lib.py
Client
__init__.py
--- main.py
Browser 1
__init__.py
--- main.py
--- B1_file1.py
--- B1_file2.py
Browser 2
__init__.py
--- main.py
--- B2_file1.py
--- B2_file2.py
Server
__init__.py
--- main.py
Server Object 1
__init__.py
--- main.py
--- SO1_file1.py
--- SO1_file2.py
Server Object 2
__init__.py
--- main.py
--- SO2_file1.py
--- SO2_file2.py
Each main.py file is used to call each downstream main.py file be it as
a thread or once only run. The beauty of this idea is that if I want
to completely change say Server Object 2, I can do whatever I like
within the downstream directory as the only outside call is via an
external reference to the main.py file.

If you have a mind, you could have a library file of common apps held
within the bin directory that you can make available to all downstream
modules.

As I say - it only works if you can logically split your program into
independent blocks.

If there is a more Pythonesque way of doing this I be grateful if
someone could share.

HTH

Oct 19 '05 #2
TokiDoki wrote:
At first, I had all of my files in one single directory, but now, with
the increasing number of files, it is becoming hard to browse my
directory. So, I would want to be able to divide the files between 8
directory, according to their purpose. The problem is that it breaks
the 'import's between my files. And besides,AFAIK, there is no
easy way to import a
file that is not in a subdirectory of the current file


Remember that the directory where you start the toplevel script is always
included in the sys.path. This means that you can have your structure like
this:

main.py
|
| - - pkg1
| - - pkg2
| - - pkg3

Files in any package can import other packages. The usual way is to do "import
pkgN" and then dereference. Within each package, you will have a __init__.py
which will define the package API (that is, will define those symbols that you
can access from outside the package).

Typically, you only want to import *packages*, not submodules. In other words,
try to not do stuff like "from pkg1.submodule3 import Foo", because this breaks
encapsulation (if you reorganize the structure of pkg1, code will break). So
you'd do "import pgk1" and later "pkg1.Foo", assuming that pkg1.__init__.p y
does something like "from submodule3 import Foo". My preferred way is to have
__init__.py just do "from submodules import *", and then each submodule defines
__all__ to specify which are its public symbols. This allow for more
encapsulation (each submodule is able to change what it exports in a
self-contained way, you don't need to modify __init__ as well).

Moreover, the dependence graph between packages shouldn't have loops. For
instance, if pkg3 uses pkg1, pkg1 shouldn't use pkg3. It makes sense to think
of pkg1 as the moral equivalent of a library, which pkg3 uses.
--
Giovanni Bajo
Oct 19 '05 #3
If I understand you right you need a concept in which you can put the files
of your project where you want, i.e. restructure the nesting of directories
storing your scripts without the problem of breaking the import statements.
This will give you not a solution to any problem you maybe have with
managing a large package but is a problem in itself, so I see that you have
a huge number of options to choose from:

1. best is you get a better structure of the project in your mind, so you
don't need frequent changes by resorting your modules into a various
directories (flat is better than nested). Set a size limit of a single
Python script it must reach before you split it into two separate files, so
that the number of your files decreases to a number you can handle with ease
(this depends on how good you are at it).

2. create e.g. an HTML file from which you can view, edit, execute the
scripts you have. In this file you create your nested structure and have
this way the overview you need. The Python files remain in one directory
storing the entire project. This allows you to keep together what belongs
together, but to work with it as the single files were spread over many
categories (I mean directories :).

3. create a Python script which automatically creates an HTML file giving
you the overview of all your scripts from walking the directories your
project files are stored below and put in each of your scripts a function
which on call returns the actual full path of passed name of the file to
import and updates the Python PATH, so that it can be found (e.g
updatePATH('nam eOfTheModuleToI mport')

4. start to choose from many tools for project management which will keep
you busy for a while and away from the actual work on your project ...

None of these above can solve the supposed actual problem, that the project
grows beyond the capacity to manage it in mind. I know about this limitation
and I have seen programmers failing on projects with increasing size of
them. Usually the dependencies between the single modules in large projects
get more and more complicated. Each next added module making the project
size larger must work with the entire already existing ones and after a
given size of project is reached it becomes really hard (and for some just
impossible) to add a new module to it making the necessary modifications to
existing ones in order to gain the new functionality. Such modifications
result usually in breaks in other parts of the project and in my eyes no
management system can really help in anticipating it at that point. Every
programmer seems to have a limit of a project size he can manage and in my
opinion no tool for structuring it or getting a better overview is of real
help here. The core of the problem is lack of skills required for managing
large projects and it seems, that not everyone who managed to learn to
program is able to acquire this skills necessary to increase the size of a
project over the by his intellectual capacity given limits. That is why
programmer need to work in teams and have a very good manager able to
coordinate their efforts in order to succeed in creating large applications.
In really large projects it is sometimes necessary to work half a year or
more full time on the existing library of code before becoming capable of
writing a single line of additional one.

Publish your code, let others work with it, try to get responses on it and
you will probably get much more out of this than by asking for ways of
managing it.

I am curious if others on this newsgroup agree with what I said above, so I
would be glad to hear about it.

Claudio

"TokiDoki" <ne*********@wa nadoo.fr> schrieb im Newsbeitrag
news:dj******** **@apollon.grec .isp.9tel.net.. .
Hello there,

I have been programming python for a little while, now. But as I am
beginning to do more complex stuff, I am running into small organization
problems.
It is possible that what I want to obtain is not possible, but I would
like the advice of more experienced python programmers.

I am writing a relatively complex program in python that has now around
40 files.
At first, I had all of my files in one single directory, but now, with
the increasing number of files, it is becoming hard to browse my directory. So, I would want to be able to divide the files between 8 directory,
according to their purpose. The problem is that it breaks the 'import's
between my files. And besides,AFAIK, there is no easy way to import a
file that is not in a subdirectory of the current file (I suppose I
could adjust the os.path in every file, but that seems not very elegant
to me).
I thought about turning the whole into a package. But I have to change
every 'import module_name' into
'from package_name.su b_directory_nam e import module_name'
which is a little bit time consuming and not very flexible (if I change
my mind about the subdirectory a file belongs to, I will have to track
again every import to correct them)

So, basically, here is the point: is there an 'elegant' way to keep my
files in separate directory and still be able to import between them
with simple 'import filename'?

And if not, what would be the standard way to organize numerous python
files belonging to the same project?

Thank you,

TokiDoki

Oct 19 '05 #4
Maybe looking at the todays thread ["dynamical" importing] can be helpful
also here.

Claudio
P.S. Below a copy of one of the responses:
:
Joerg Schuster wrote:
I need to import modules from user defined paths. I.e. I want to do
something like:

module_dir = sys.argv[1]

my_path = os.path.join(mo dule_dir, 'bin', 'my_module')

from my_path import my_object

Obviously, it doesn't work this way. How would it work?
some alternatives:

- if you want the modules to remain imported:

try:
sys.path.insert (0, os.path.join(mo dule_dir, "bin"))
module = __import__("my_ module")
finally:
del sys.path[0]
object = module.my_objec t

- if you're only interested in the object:

namespace = {}
execfile(os.pat h.join(module_d ir, "bin", "my_module" + ".py"),
namespace)
object = namespace["my_object"]

</F>
"TokiDoki" <ne*********@wa nadoo.fr> schrieb im Newsbeitrag
news:dj******** **@apollon.grec .isp.9tel.net.. . Hello there,

I have been programming python for a little while, now. But as I am
beginning to do more complex stuff, I am running into small organization
problems.
It is possible that what I want to obtain is not possible, but I would
like the advice of more experienced python programmers.

I am writing a relatively complex program in python that has now around
40 files.
At first, I had all of my files in one single directory, but now, with
the increasing number of files, it is becoming hard to browse my directory. So, I would want to be able to divide the files between 8 directory,
according to their purpose. The problem is that it breaks the 'import's
between my files. And besides,AFAIK, there is no easy way to import a
file that is not in a subdirectory of the current file (I suppose I
could adjust the os.path in every file, but that seems not very elegant
to me).
I thought about turning the whole into a package. But I have to change
every 'import module_name' into
'from package_name.su b_directory_nam e import module_name'
which is a little bit time consuming and not very flexible (if I change
my mind about the subdirectory a file belongs to, I will have to track
again every import to correct them)

So, basically, here is the point: is there an 'elegant' way to keep my
files in separate directory and still be able to import between them
with simple 'import filename'?

And if not, what would be the standard way to organize numerous python
files belonging to the same project?

Thank you,

TokiDoki

Oct 19 '05 #5
I, too have often come up against the inconvenience of creating
libraries that I want to reuse but that do not want to incorporate into
the Python library. I came up with this Python library addition to
automagically add the directory of where a module resides that I want
to import to the system path, so that I can simply add the module by
name without needing to worry about anything else.

The large advantage of this is that I can now import from adjacent
directories rather than subordinate ones. For best results in large
projects with deep tree structure, the paths of the importing script
and the imported module should have at least 3 levels in common. I do
not consider typing 3 levels of pathing as a hint to be a great
inconvenience.

It ain't perfect, it ain't Zen, but for me this has worked from Python
2.2 and up with very few problems for projects up to 1000 files.
Improvements, anyone?

The Eternal Squire

#----------------------relative.py---------------------
# Place in Python##\Lib
#
import sys
from inspect import getfile
from os.path import abspath
backslash = '\\'

def caller ():

'name of the calling script'

frame = sys._getframe(1 )
if not frame: return ''
frame = frame.f_back
if not frame: return ''
result = getfile (frame)
if result[-4:].lower () in ['.pyc', '.pyo']: result = filename[:-4]
+ '.py'
return abspath (result)

def append (path):

'append relative path to system module search path'
if not path: return

try:
calling_script = caller ()

top_path = path.split (backslash)[0]
where = calling_script. find (top_path + backslash)

if where < 0: return
result = calling_script [:where] + path

if result not in sys.path: sys.path.append (result)
return result
except:
return ''
#---------------------end of relative.py---------------#

#---------------------"A\B\C\more levels\importer .py"-----------------#
#prototypical code, do not try to execute

import relative
relative.append ("\A\B\C\subpat h") # no file extension!
from module import whatsit # module resides under subpath
whatsit ()

Oct 19 '05 #6
Giovanni Bajo napisa³(a):
Remember that the directory where you start the toplevel script is always
included in the sys.path. This means that you can have your structure like
this:

main.py
|
| - - pkg1
| - - pkg2
| - - pkg3

Files in any package can import other packages. The usual way is to do "import
pkgN" and then dereference. Within each package, you will have a __init__.py
which will define the package API (that is, will define those symbols that you
can access from outside the package).


How to install this structure eg. on Linux? What layout do you
recommend? It's tempting to use /opt hierarchy for installation target
(as it gives relatively much freedom within application directory), but
many administrators are reluctant to use this hierarchy and prefer more
standarized targets, such as /usr or /usr/local.

--
Jarek Zgoda
http://jpa.berlios.de/
Oct 19 '05 #7
On Oct 19, Jarek Zgoda wrote:
How to install this structure eg. on Linux? What layout do you
recommend? It's tempting to use /opt hierarchy for installation
target (as it gives relatively much freedom within application
directory), but many administrators are reluctant to use this
hierarchy and prefer more standarized targets, such as /usr or
/usr/local.


Read about (and use) the Python-provided distutils, and let it do the
work for you. In particular,
<http://www.python.org/doc/2.4.2/inst/alt-install-windows.html>
discusses installation location. The file name is a misnomer; it's
equally applicable to linux.

It allows users to easily specify the installation prefix. The
default should be the value of "sys.prefix ".

Note that your modules/packages will be installed within your python
installation under its site-packages/ directory.

--
_ _ ___
|V|icah |- lliott http://micah.elliott.name md*@micah.ellio tt.name
" " """
Oct 19 '05 #8
Micah Elliott napisa³(a):
How to install this structure eg. on Linux? What layout do you
recommend? It's tempting to use /opt hierarchy for installation
target (as it gives relatively much freedom within application
directory), but many administrators are reluctant to use this
hierarchy and prefer more standarized targets, such as /usr or
/usr/local.


Read about (and use) the Python-provided distutils, and let it do the
work for you. In particular,
<http://www.python.org/doc/2.4.2/inst/alt-install-windows.html>
discusses installation location. The file name is a misnomer; it's
equally applicable to linux.


I think that installing *application* (not misc. library) modules in
site-packages is at least stupid idea, as it makes a special filesystem
inside other, much more standarized filesystem (Windows, FHS, etc.). Why
not to do this usual way: libraries to $prefix/lib/$appname-$version,
binary to $prefix/bin, shared files to $prefix/share/$appname, etc --
and appropriately on Windows.

--
Jarek Zgoda
http://jpa.berlios.de/
Oct 19 '05 #9
Jarek Zgoda wrote:
How to install this structure eg. on Linux? What layout do you
recommend? It's tempting to use /opt hierarchy for installation target
(as it gives relatively much freedom within application directory),
but
many administrators are reluctant to use this hierarchy and prefer
more standarized targets, such as /usr or /usr/local.

I am not very experienced with Linux, but if you don't use something like
PyInstaller, you could still install the main tree somewhere like
/usr/local/myapp/ and then generate a simple script for /usr/local/bin which
adds /usr/local/myapp to sys.path[0], and "import main" to boot the
application.

I'm not sure I have answered your question though :)
--
Giovanni Bajo
Oct 19 '05 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

10
11074
by: Bruce W...1 | last post by:
I've been learning about PHP for a couple of weeks. With includes, PHP scripts, and HTML I can see where a large and complex website could easily turn in to a big hairy mess with files all over the place. Are there any adopted standards, recognized recommendations, or best practices on how all the code should be organized? I haven't found any websites that discuss this. Can anyone point me to information on this? If not then what do...
36
6424
by: Andrea Griffini | last post by:
I did it. I proposed python as the main language for our next CAD/CAM software because I think that it has all the potential needed for it. I'm not sure yet if the decision will get through, but something I'll need in this case is some experience-based set of rules about how to use python in this context. For example... is defining readonly attributes in classes worth the hassle ? Does duck-typing scale well in complex
9
2260
by: seberino | last post by:
I have been using distuils for a while and was wondering when Python Eggs (new project) is better? So basically Python Eggs precompiles and compresses binaries for you so you just have to load it to run your app? Chris
0
1242
by: tommaso.gastaldi | last post by:
I have a solution with 1 project on which I am working on say Pr1 I have another solution with another project, say Pr2. Source file in the existing Pr2 are organized within several nested folders. When I work on the new project Pr1 I would like to have all the source files of Pr2 ( * no DLL !! * ) available within Pr1, without copying them, so that any change made within Pr1 is also present in Pr2.
12
2322
by: Jchick | last post by:
Boy, this should be a simple bit of code but I can't figure out how to make it happen. I have a CSV file shows up in a directory that has 4 fields that need to be printed on labels. Each line of the CSV looks something like this: AcctNo, Name, junk, junk, Address, junk, PhoneNo I need to read each line of the CSV and print them to Avery Label stock
25
2372
by: Eric | last post by:
Hello, after reading some of the book Programming Python it seems that python is something I would like to delve deeper into. The only thing is, I have no idea what I should try and write. So I was hoping that someone here could help point me to a group/project that would be a good starting place for a person with limited python knowledge, but that is willing to learn whatever is necessary. I'm hoping that with a goal I can start to learn...
0
1161
by: Dudeja, Rajat | last post by:
Hi, I'm learning Python to write a GUI application using Tkinter & Tk. I've evaluated Eclipse and Pydev. With the help of Fabio Zadrozny I successfully installed Eclipse and PyDev. Thanks. Now, I'm learning Eclipse and PyDev. And my problem is that I don't have an understanding of how the code in
2
2483
by: Fredrik Lundh | last post by:
Dudeja, Rajat wrote: A Python program consists of a script file (the py file you run to start the program), and usually one or more additional module files (py files that you import). The latter can be organized in packages, where appropriate. There's also a search path (sys.path) that you can modify in various ways (including from within the program) if you want to fetch modules from different locations.
0
10936
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10609
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10669
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10303
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9448
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
1
7853
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
5695
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
5882
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
2
4085
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.