473,320 Members | 2,133 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Python for Large Projects

Hello all. :)

I've been a long time Python fan, and have fairly recently (with the
support of a coworker who surprised me by mentioning my pet language one
day) convinced my company to begin the colossal task of basically rewriting
all of our software in Python. Woohoo.
Previously we used a few different development environments, mostly
Borland, for different products in our 'system' of thick clients sort of
operating with eachother as they dug against a database. The whole thing is
woefully out of date at this point, and platform-dependant, so we needed to
switch to something.. and Python won. Yay.
In the new platform, we'll be building a distributed system with CORBA
(omniORB + omniORBpy, etc) with Python making up most of our development for
the new edition of our software. Woohoo. Much fun.

My question is not 'how suitable is Python for large projects?', but
instead more, 'Do you have any advice for large projects with Python?'.

For example, our system will be made up of numerous applications --
three or four 'servers', multiple clients, all talking to eachother, sharing
a great deal of code. Any suggestions for organization? Our current plan is
to put everything under the 'site-packages' tree of a Python installation we
provide/control, with a fairly deeply nested tree:

apt
apt\library
apt\library\corba.py
apt\library\database\*
apt\library\controls\*
apt\application\globalapp1\
apt\application\globalapp2\
apt\system\editorial\application\client\
apt\system\editorial\application\client2\
apt\system\editorial\library\elementgrid.py

.... etc. You get the idea, I hope. We have apt.library where we store all of
our 'global', 'shared' libraries that everything can make use of, and
apt.application with some global, shared apps. Then we have 'systems', which
are a collection of applications and libraries in a single domain. These
mostly get sold as a single product, really. Etc, etc.

I end up doing a lot of:

from apt.library.corba import *
from apt.library.database import Connection
from apt.library import machine
from apt.system.editorial.library.elementgrid import ElementGrid

... And such. When PEP328 gets in, this may become more attractive, but
occasionally I'm alarmed by the depth of the whole thing. But I also want to
keep stuff logical and separated.

And how about distribution? Freezing and such won't work, because of the
shared nature of most of this code. Currently I'm probably planning on
looking into finding the build scripts for python, win32all, and wxPython
and seeing if I can combine them into one single install. Don't know if
that's the best thing to do or not, but expecting my customers to install
this stuff... is expecting too much of them.

For example: Traditionally my company has avoided installing software on
a users local machine. Instead we install to a file server and have everyone
run the software from there. A bit of a performance hit but the advantage of
administration is very important, because our customers do not necessarily
have a very strong IT staff. There is some concern that if we do this same
thing with Python, the performance hit of loading an interpreted environment
over the shared filesystem will be prohibitive. Some basic testing has made
this seem to be the case.

hmm. That's all I have for now. I think. :)

Thanks in advance.

--Stephen
Jul 18 '05 #1
7 2372
Ixokai wrote:
There is some concern that if we do this same
thing with Python, the performance hit of loading an interpreted environment
over the shared filesystem will be prohibitive. Some basic testing has made
this seem to be the case.


Just focusing on this one point: it doesn't make sense to me that
this would be the case, and in my experience it is not. The
interpreter (python23.dll for example on Win32) is less than 1MB
and that's basically _tiny_ these days. We always had a single
shared instance of Python and never noticed more than a small
fraction of second delay compared to loading it locally. This
was a network with only about fifteen developers as users, and
100MBps throughout, but I would think almost any program these
days is going to be 1MB or more, so I can't see that Python has
any disadvantage in this area.

-Peter
Jul 18 '05 #2
Peter Hansen wrote:
Ixokai wrote:
There is some concern that if we do this same
thing with Python, the performance hit of loading an interpreted
environment
over the shared filesystem will be prohibitive. Some basic testing has
made
this seem to be the case.

Just focusing on this one point: it doesn't make sense to me that
this would be the case, and in my experience it is not. The
interpreter (python23.dll for example on Win32) is less than 1MB
and that's basically _tiny_ these days. We always had a single
shared instance of Python and never noticed more than a small
fraction of second delay compared to loading it locally. This
was a network with only about fifteen developers as users, and
100MBps throughout, but I would think almost any program these
days is going to be 1MB or more, so I can't see that Python has
any disadvantage in this area.

-Peter

I'm a newbie so take what I say with a grain of salt!
In the VFP (visual foxpro) world the runtime environment is loaded on
the local computer (client) and the program is often loaded on to the
server. I'm wondering if there would be any advanage to placing the
python runtime on the clients along with the standard modules. Then
placing the program code on to a server where all the clients/users
would access the code. It would seem that the advanage would be that
program load time would about the same but the excutions would be faster
because of the local runtime. Can this even been done? Would it be faster?

John
Jul 18 '05 #3
On Dom 16 Mai 2004 12:25, john fabiani wrote:
I'm a newbie so take what I say with a grain of salt!
In the VFP (visual foxpro) world the runtime environment is loaded on
the local computer (client) and the program is often loaded on to the
server. I'm wondering if there would be any advanage to placing the
python runtime on the clients along with the standard modules. Then
placing the program code on to a server where all the clients/users
would access the code. It would seem that the advanage would be that
program load time would about the same but the excutions would be faster
because of the local runtime. Can this even been done? Would it be
faster?


If I understood you correctly, this is what we do on our clients. It also
makes it easier to make bugfixes, updates, etc. in our code.

--
Godoy. <go***@ieee.org>
Jul 18 '05 #4
Jorge Godoy wrote:
On Dom 16 Mai 2004 12:25, john fabiani wrote:

I'm a newbie so take what I say with a grain of salt!
In the VFP (visual foxpro) world the runtime environment is loaded on
the local computer (client) and the program is often loaded on to the
server. I'm wondering if there would be any advanage to placing the
python runtime on the clients along with the standard modules. Then
placing the program code on to a server where all the clients/users
would access the code. It would seem that the advanage would be that
program load time would about the same but the excutions would be faster
because of the local runtime. Can this even been done? Would it be
faster?

If I understood you correctly, this is what we do on our clients. It also
makes it easier to make bugfixes, updates, etc. in our code.

So it can be done. I'll have to learn a little more about path-ing.
Thanks
John
Jul 18 '05 #5

"Peter Hansen" <pe***@engcorp.com> wrote in message
news:DL********************@powergate.ca...
Ixokai wrote:
There is some concern that if we do this same
thing with Python, the performance hit of loading an interpreted environment over the shared filesystem will be prohibitive. Some basic testing has made this seem to be the case.


Just focusing on this one point: it doesn't make sense to me that
this would be the case, and in my experience it is not. The
interpreter (python23.dll for example on Win32) is less than 1MB
and that's basically _tiny_ these days. We always had a single
shared instance of Python and never noticed more than a small
fraction of second delay compared to loading it locally. This
was a network with only about fifteen developers as users, and
100MBps throughout, but I would think almost any program these
days is going to be 1MB or more, so I can't see that Python has
any disadvantage in this area.


These are GUI programs and we've decided to use wxPython, so that may be
adding to it. I will do more testing to see if it was just a fluke the last
time I tried it out. It might have been :)

--Stephen
Jul 18 '05 #6
Ixokai wrote:
"Peter Hansen" <pe***@engcorp.com> wrote:
Ixokai wrote:
There is some concern that if we do this same thing with Python,
the performance hit of loading an interpreted environment over
the shared filesystem will be prohibitive. Some basic testing
has made this seem to be the case.


Just focusing on this one point: it doesn't make sense to me that
this would be the case, and in my experience it is not....


These are GUI programs and we've decided to use wxPython, so that
may be adding to it. I will do more testing to see if it was just
a fluke the last time I tried it out. It might have been :)


One thing to investigate is file openings. If you combine all of
python (and in your case wxPython's) .pyc files into a single zip
wish an actual directory structure matching the packages you want to
use, you will only have to open the zip file once, rather than pull
each module in in several file operations.
--
-Scott David Daniels
Sc***********@Acm.Org
Jul 18 '05 #7
Ixokai wrote:
Hello all. :)

I've been a long time Python fan, and have fairly recently (with the
support of a coworker who surprised me by mentioning my pet language one
day) convinced my company to begin the colossal task of basically rewriting
all of our software in Python. Woohoo.
Previously we used a few different development environments, mostly
Borland, for different products in our 'system' of thick clients sort of
operating with eachother as they dug against a database. The whole thing is
woefully out of date at this point, and platform-dependant, so we needed to
switch to something.. and Python won. Yay.
In the new platform, we'll be building a distributed system with CORBA
(omniORB + omniORBpy, etc) with Python making up most of our development for
the new edition of our software. Woohoo. Much fun.

My question is not 'how suitable is Python for large projects?', but
instead more, 'Do you have any advice for large projects with Python?'.

For example, our system will be made up of numerous applications --
three or four 'servers', multiple clients, all talking to eachother, sharing
a great deal of code. Any suggestions for organization? Our current plan is
to put everything under the 'site-packages' tree of a Python installation we
provide/control, with a fairly deeply nested tree:

apt
apt\library
apt\library\corba.py
apt\library\database\*
apt\library\controls\*
apt\application\globalapp1\
apt\application\globalapp2\
apt\system\editorial\application\client\
apt\system\editorial\application\client2\
apt\system\editorial\library\elementgrid.py

... etc. You get the idea, I hope. We have apt.library where we store all of
our 'global', 'shared' libraries that everything can make use of, and
apt.application with some global, shared apps. Then we have 'systems', which
are a collection of applications and libraries in a single domain. These
mostly get sold as a single product, really. Etc, etc.

I end up doing a lot of:

from apt.library.corba import *
from apt.library.database import Connection
from apt.library import machine
from apt.system.editorial.library.elementgrid import ElementGrid

... And such. When PEP328 gets in, this may become more attractive, but
occasionally I'm alarmed by the depth of the whole thing. But I also want to
keep stuff logical and separated. Well you don't neccessarily have to have everything under apt... You
just need to have unique package names.
Also the cleaner your system design the less you should have to refer to
bits inside other packages.
And how about distribution? Freezing and such won't work, because of the
shared nature of most of this code. Currently I'm probably planning on
looking into finding the build scripts for python, win32all, and wxPython
and seeing if I can combine them into one single install. Don't know if
that's the best thing to do or not, but expecting my customers to install
this stuff... is expecting too much of them.


On the wxPython users list we are planning something called wxPRE which
would be an environment pre-setup with Python and wxPython. You may want
to help set this up as it is very similar to your requirements.
See http://wiki.wxpython.org/index.cgi/wxPRE

David
Jul 18 '05 #8

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

42
by: Bicho Verde | last post by:
I have now free time and money to do what I want :-) I have some basic skills in programming (C, Pascal, Macromedia Actionscript) but don't know exactly what to do in the world of programming. And...
23
by: assaf__ | last post by:
Hello, I am beginning to work on a fairly large project and I'm considering to use python for most of the coding, but I need to make sure first that it is reliable enough. I need to make sure...
42
by: Fred Ma | last post by:
Hello, This is not a troll posting, and I've refrained from asking because I've seen similar threads get all nitter-nattery. But I really want to make a decision on how best to invest my time....
9
by: limor | last post by:
Hi, I am considering using Python in a new testing tool application we intend to build for out product. I must get references before starting develope in this language , since although lots of...
36
by: Andrea Griffini | last post by:
I did it. I proposed python as the main language for our next CAD/CAM software because I think that it has all the potential needed for it. I'm not sure yet if the decision will get through, but...
1
by: Adriaan Renting | last post by:
I think the point you want to make is that Python needs vastly less lines of code as a similar application written in C++. I think Python might on average be 50-60% of comparable C++ code, but not...
10
by: TokiDoki | last post by:
Hello there, I have been programming python for a little while, now. But as I am beginning to do more complex stuff, I am running into small organization problems. It is possible that what I...
267
by: Xah Lee | last post by:
Python, Lambda, and Guido van Rossum Xah Lee, 2006-05-05 In this post, i'd like to deconstruct one of Guido's recent blog about lambda in Python. In Guido's blog written in 2006-02-10 at...
162
by: Sh4wn | last post by:
Hi, first, python is one of my fav languages, and i'll definitely keep developing with it. But, there's 1 one thing what I -really- miss: data hiding. I know member vars are private when you...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.