Is Python suitable for a huge, enterprise size app?

[john67]

The company I work for is about to embark on developing a commercial
application that will cost us tens-of-millions to develop. When all is
said and done it will have thousands of business objects/classes, some
of which will have hundreds-of-thousands of instances stored in a DB.
Our clients will probably have somewhere between 50-200 users working
on the app during the day, possibly in mutiple offices, and then a
large number of batch processes will have to run each night. It will
also need to have a web interface for certain users. It needs to be
robust, easy to maintain, and able to be customized for each client.

Right now it looks like Java is the language of choice that the app
will be developed in. However, I have been looking and reading a lot
about Python recently and it seems to me that Python could handle it.
The big attraction to me is the developer productivity. It seems that
if Python can handle it, then we could gain a huge savings by using
Python instead of Java from a productivity standpoint alone.

So, given the very general requirements in the first paragraph, do you
think that Python could handle it? If anyone has direct experience
developing large apps in Python, I would appreciate your insight.
Based on the responses I get, I am planning on writing a proposal to my
management to consider Python instead of Java.

Thanks,
John

[bplumhoff]

The answer seems to be yes. See http://www.artima.com/intv/speed.html

HTH,
Bernd

[keirr]

> So, given the very general requirements in the first paragraph, do
you

think that Python could handle it? If anyone has direct experience
developing large apps in Python, I would appreciate your insight.

I wouldn't, especially[1] if your thousands of business objects get
allocated/deallocated as the system runs. Currently python's memory
usage can grow rapidly (from the perspective of the o/s) when large
numbers of objects are repeatedly created and freed.

On the other hand, you could use a combination of python and C++
(perhaps using boost's python wrapping code). The ability to create
C/C++ modules is a "get out of jail free card" - if you did manage to
convince your company to go with python, and it fell short in some
area, a custom C module could be a handy trick.

Perhaps you could compare your project to existing large python
projects; like Zope maybe?

[1] if the objects in the system are static, rather than dynamic, of
course this is not an issue. My "I wouldn't" comes from the fact that
your requirements include "lots of object" as a feature, and no word on
how they will be used.

Good luck anyway :-)
Keir.

[Larry Bates]

I don't see anything listed that would eliminate Python. You
may want to take a quick look at some of the rather large
applications that have been done with Zope (written in Python).
Zope might even be a good application platform to build upon
(can't say for sure, not enough detail about application).
Even if you don't want to use all of Zope the ZODB is a well
tested persistent Python object store that might prove
helpful.

There are many other projects supporting WAY more than 200
users that are written in Python.

-Larry Bates

john67 wrote:

The company I work for is about to embark on developing a commercial
application that will cost us tens-of-millions to develop. When all is
said and done it will have thousands of business objects/classes, some
of which will have hundreds-of-thousands of instances stored in a DB.
Our clients will probably have somewhere between 50-200 users working
on the app during the day, possibly in mutiple offices, and then a
large number of batch processes will have to run each night. It will
also need to have a web interface for certain users. It needs to be
robust, easy to maintain, and able to be customized for each client.

Right now it looks like Java is the language of choice that the app
will be developed in. However, I have been looking and reading a lot
about Python recently and it seems to me that Python could handle it.
The big attraction to me is the developer productivity. It seems that
if Python can handle it, then we could gain a huge savings by using
Python instead of Java from a productivity standpoint alone.

So, given the very general requirements in the first paragraph, do you
think that Python could handle it? If anyone has direct experience
developing large apps in Python, I would appreciate your insight.
Based on the responses I get, I am planning on writing a proposal to my
management to consider Python instead of Java.

Thanks,
John

[john67]

Would the allocation/deallocation memory usage issue be different than
it would be with Java? Both Python and Java have automatic garbage
collection, correct? Is Python's not as effective as Java's? I think
the memory issues for the two languages would be similar in this area,
but maybe I am wrong.

John

[Peter Dembinski]

"keirr" <ke***********@ gmail.com> writes:

[snap]

I wouldn't, especially[1] if your thousands of business objects get
allocated/deallocated as the system runs. Currently python's memory
usage can grow rapidly (from the perspective of the o/s) when large
numbers of objects are repeatedly created and freed.
Isn't it true that in recent Python releases, one may replace the
default GC with custom one?
On the other hand, you could use a combination of python and C++
(perhaps using boost's python wrapping code). The ability to create
C/C++ modules is a "get out of jail free card" - if you did manage
to convince your company to go with python, and it fell short in
some area, a custom C module could be a handy trick.

In fact, it is not a trick -- combining C with Python is fairly
approved way of speeding up the app :)

--
http://www.peter.dembinski.prv.pl

[keirr]

>> I wouldn't, especially[1] if your thousands of business objects get

allocated/deallocated as the system runs. Currently python's memory
usage can grow rapidly (from the perspective of the o/s) when large
numbers of objects are repeatedly created and freed.

Isn't it true that in recent Python releases, one may replace the
default GC with custom one?

Good point. Indeed, since you have the source code you can change
anything you like. My warning was aimed at a specific situation, and
was meant to be a 'keep this in mind' comment (I phrased it as "don't
use python" really because I expected the majority of the responses to
be "do use python", and thought a bit of balance would be helpful).

As for a custom garbage collector, there has been a little discussion
about the gc code (on this group). My feeling is, if it was easy to
make it better someone would have done it already. Note I don't say
it's impossible (particularly with the resources the op mentioned),
just not straight-forward.

Of course, I wouldn't use Java, but that's another story '-)

Cheers,

Keir.

[Dave Brueck]

john67 wrote:

The company I work for is about to embark on developing a commercial
application that will cost us tens-of-millions to develop. [snip] Right now it looks like Java is the language of choice that the app
will be developed in. However, I have been looking and reading a lot
about Python recently and it seems to me that Python could handle it.
The big attraction to me is the developer productivity. It seems that
if Python can handle it, then we could gain a huge savings by using
Python instead of Java from a productivity standpoint alone.

So, given the very general requirements in the first paragraph, do you
think that Python could handle it?

Yes.

Python has good database support, it works well on a wide range of platforms,
and it's great at tying together different processes, machines, etc. - for
example, it's fairly easy to get Python to access C code, dynamic libraries,
system APIs, and external programs. It's easier to test than Java code as well,
and the overal cost of change is lower.

The concise nature of the language means that what would otherwise be a "huge"
enterprise app in Java might just be a "large" enterprise app in Python - the
project will not reach that unwieldy size as quickly, if ever.

The Google archives of this group have this topic covered in various ways many
times over - IMO Python is great for smallish apps, but its advantage over e.g.
Java actually *increases* with the size of the app.

If you go the Python route, two of the main obstacles will be:

1) mindshare - getting people on board, getting them to overcome biases one way
or another, convincing them to really take the time to come up to speed on Python.

2) reducing the complexity of what you think you need to build. I don't know how
to better describe this, but in Java, for example, I'd have whole bunches custom
classes to do various tasks but the Python equivalent collapsed into a single
module. Anyway, I predict that it'll take you some time to convince yourself
that you simply won't need to build all of the same components as you otherwise
would, or that they'll often be vastly simpler.

-Dave

[Cameron Laird]

In article <11************ **********@f14g 2000cwb.googleg roups.com>,
john67 <te*******@gmai l.com> wrote:

The company I work for is about to embark on developing a commercial
application that will cost us tens-of-millions to develop. When all is
said and done it will have thousands of business objects/classes, some
of which will have hundreds-of-thousands of instances stored in a DB.
Our clients will probably have somewhere between 50-200 users working
on the app during the day, possibly in mutiple offices, and then a
large number of batch processes will have to run each night. It will
also need to have a web interface for certain users. It needs to be
robust, easy to maintain, and able to be customized for each client.

Right now it looks like Java is the language of choice that the app
will be developed in. However, I have been looking and reading a lot
about Python recently and it seems to me that Python could handle it.
The big attraction to me is the developer productivity. It seems that
if Python can handle it, then we could gain a huge savings by using
Python instead of Java from a productivity standpoint alone.

So, given the very general requirements in the first paragraph, do you
think that Python could handle it? If anyone has direct experience
developing large apps in Python, I would appreciate your insight.
Based on the responses I get, I am planning on writing a proposal to my
management to consider Python instead of Java.

[Steve M]

This thread:

http://mail.python.org/pipermail/pyt...ry/051255.html

discusses the problem with memory allocation in CPython. Apparently
CPython is not good at, or incapable of, releasing memory back to the
operating system. There are ways to compensate for this. I guess the
comment about C modules was meant as one way to do so, either by
reducing memory requirement in the first place (C data structures are
more compact than Python) or else by allocating and freeing memory
wholly in the C module, which perhaps does work as expected.

The web page for Evan Jones does not obviously indicate the status of
the improved memory allocator patch he was working on. I wonder if it
is coming along.

Incidentally, does anyone know the prospects for CPython to be made
stackless? Say in 2.5 or 2.9? Or will that always be an independent
project?