python scalability

[Tim Mitchell]

Hi All,

I work on a desktop application that has been developed using python and
GTK (see www.leapfrog3d.com). We have around 150k lines of python code
(and 200k+ lines of C). We also have a new project manager with a C#
background who has deep concerns about the scalability of python as our
code base continues to grow and we are looking at introducing more
products. I am looking for examples of other people like us (who write
desktop apps in python) with code bases of a similar size who I can
point to (and even better talk to) to help convince him that python is
scalable to 300+ lines of code and beyond. I have looked at the python
success stories page and haven't come up with anyone quite like us.
One of my project managers questions is: "Are we the only company in the
world with this kind and size of project?"
I want to say no, but am having trouble convincing myself, let alone him.

If you are involved in this kind of thing please get in touch with me.

Thanks,
Tim

[Mike Hansen]

I have looked at the python

success stories page and haven't come up with anyone quite like us.
One of my project managers questions is: "Are we the only company in the
world with this kind and size of project?"
I want to say no, but am having trouble convincing myself, let alone him.

If you are involved in this kind of thing please get in touch with me.

While Sage ( http://www.sagemath.org ) is more of a library than an
application, it is approximately 350k lines of Python and Cython (in
roughly a 2 to 1 ratio). I think it has scaled surprisingly well.

--Mike

[Bruno Desthuilliers]

Tim Mitchell a écrit :

Hi All,

I work on a desktop application that has been developed using python and
GTK (see www.leapfrog3d.com). We have around 150k lines of python code
(and 200k+ lines of C). We also have a new project manager with a C#
background who has deep concerns about the scalability of python as our
code base continues to grow and we are looking at introducing more
products. I am looking for examples of other people like us (who write
desktop apps in python) with code bases of a similar size who I can
point to (and even better talk to) to help convince him that python is
scalable to 300+ lines of code and beyond.

As far as I'm concerned, I'd say that if you didn't feel anything wrong
nor spot any problem so far working on a 150 klocs project, there's just
no need to worry. FWIW, a C# port would probably requires *way* much
than 150 klocs.

I have looked at the python
success stories page and haven't come up with anyone quite like us. One
of my project managers questions is: "Are we the only company in the
world with this kind and size of project?"

Doing a quick count using wc -l (so the following numbers include
comments and blank lines), Zope 2.8 is about 418 klocs, Plone 3 about
259 klocs. Which, for the full stack, makes about 677 klocs. Even if you
estimate a 50/50 ratio between real locs and comments/blank lines, this
still makes something like 338 klocs.

Is that enough ?-)

[Michele Simionato]

On Jul 10, 6:32*am, Tim Mitchell <[e-mail]:

Hi All,

I work on a desktop application that has been developed using python and
GTK (seewww.leapfrog3d.com). *We have around 150k lines of python code
(and 200k+ lines of C).

We have bigger numbers than yours here (although not for a desktop
application)
and of course we have the problems of a large size application, but
they have
nothing to do with Python. The real problem are sociological, not
language-related.
Essentially, if a project takes 10+ years and 10+ people, with most of
the people
new, you have an issue, but this is independent from the language.
Python is helping
us at least because it is readable and the situation would be probably
be worse with
another language. But as I said the software development practices
used are more
important than the language in this context.

[Jeffrey Froman]

Tim Mitchell wrote:

One of my project managers questions is: "Are we the only company in the
world with this kind and size of project?"

I can't provide a bigger success story personally (my largest project is
currently about 15k lines of code, eminently manageable by one person.) But
Google comes to mind. Just a feeling I get from quotes like:

"Python is big at Google ... being used for everything from build tools to
managing ads."
-- Guido Van Rossum
http://www.artima.com/weblogs/viewpo...?thread=143947

"When programming in python, one should design for scalability. If you
design properly, the you should be able to just throw more machines at the
problem and your app should scale fine."
-- Greg Stein's PyCON keynote, paraphrased by Matt Harrison
http://panela.blog-city.com/python_a...n__sdforum.htm

Maybe that's not exactly the sort of scalability you're referring to, but
certainly Goolge is colossal in its deployment. I have a hard time
imagining that scalability based on "N lines of code" is going to be any
better in any other language.
Jeffrey

[Larry Bates]

Tim Mitchell wrote:

Hi All,

I work on a desktop application that has been developed using python and
GTK (see www.leapfrog3d.com). We have around 150k lines of python code
(and 200k+ lines of C). We also have a new project manager with a C#
background who has deep concerns about the scalability of python as our
code base continues to grow and we are looking at introducing more
products. I am looking for examples of other people like us (who write
desktop apps in python) with code bases of a similar size who I can
point to (and even better talk to) to help convince him that python is
scalable to 300+ lines of code and beyond. I have looked at the python
success stories page and haven't come up with anyone quite like us. One
of my project managers questions is: "Are we the only company in the
world with this kind and size of project?"
I want to say no, but am having trouble convincing myself, let alone him.

If you are involved in this kind of thing please get in touch with me.

Thanks,
Tim

It is not clear what "scalability" means to a desktop application? When maximum
calculation speeds are required, you can drop into C libraries from Python with
ease. If this is about making use of multiple cores on a machine this link
(http://wiki.python.org/moin/ParallelProcessing) covers most of the solutions
that can be implementing in Python.

Another poster has suggested that you would need 1500K lines of C to replace
300K lines of Python (and he may be being generous to C). Projects are about
more than raw speed: maintainability, speed of adding features, etc. Python
wins those races in most situations. I'm sure that hand coded assembler would
run fastest of all, but little application software is written that way these days.

-Larry

[Tim Mitchell]

Thanks for all the replies - they have all been helpful.

On reflection I think our problems are probably design and people related.

Cheers,
Tim

Michele Simionato wrote:

On Jul 10, 6:32 am, Tim Mitchell <[e-mail]:

>Hi All,

I work on a desktop application that has been developed using python and
GTK (seewww.leapfrog3d.com). We have around 150k lines of python code
(and 200k+ lines of C).

We have bigger numbers than yours here (although not for a desktop
application)
and of course we have the problems of a large size application, but
they have
nothing to do with Python. The real problem are sociological, not
language-related.
Essentially, if a project takes 10+ years and 10+ people, with most of
the people
new, you have an issue, but this is independent from the language.
Python is helping
us at least because it is readable and the situation would be probably
be worse with
another language. But as I said the software development practices
used are more
important than the language in this context.
--
http://mail.python.org/mailman/listinfo/python-list

[=?ISO-8859-1?Q?Gerhard_H=E4ring?=]

Tim Mitchell wrote:

Thanks for all the replies - they have all been helpful.

On reflection I think our problems are probably design and people related.

I strongly agree. "Scalability" is becoming a buzzword lately, which is
meaningless unless qualified what exactly is meant.

It's overused wrt technologies, programming languages, tools, etc.

But in other interesting areas people think much less about "scalability".

Will the current software development process work with a team of 5, 10,
20, 50 team members? Does it "scale" here?

Will the software development process still work with an increased
turnover rate? Does it "scale"?

Will the way of doing things still work with an increasingly aging codebase?

-- Gerhard

[Dan Stromberg]

On Thu, 10 Jul 2008 16:32:38 +1200, Tim Mitchell wrote:

Hi All,

I work on a desktop application that has been developed using python and
GTK (see www.leapfrog3d.com). We have around 150k lines of python code
(and 200k+ lines of C). We also have a new project manager with a C#
background who has deep concerns about the scalability of python as our
code base continues to grow and we are looking at introducing more
products. I am looking for examples of other people like us (who write
desktop apps in python) with code bases of a similar size who I can
point to (and even better talk to) to help convince him that python is
scalable to 300+ lines of code and beyond. I have looked at the python
success stories page and haven't come up with anyone quite like us. One
of my project managers questions is: "Are we the only company in the
world with this kind and size of project?" I want to say no, but am
having trouble convincing myself, let alone him.

If you are involved in this kind of thing please get in touch with me.

Thanks,
Tim

It sounds less like concerns about python, and more about "nobody ever
got fired for going microsoft" (except they have).

You might focus not just on large python projects, but also major
companies with substantial commitments to python.

That said, not all python implementations scale that well with
multithreading. It sounds like multithreaded apps are fine for single
CPU machines, and fine on multi-CPU boxes if the app is more I/O bound
than CPU bound - but that's all a matter of implementation, not language
definition, and some python implementations purportedly do well anyway.

It might not hurt to mention IronPython either, since it's a python from
microsoft.

[Paul Hankin]

On Jul 10, 5:32*am, Tim Mitchell <[e-mail]:

I work on a desktop application that has been developed using python and
GTK (seewww.leapfrog3d.com). *We have around 150k lines of python code
(and 200k+ lines of C). *We also have a new project manager with a C#
background who has deep concerns about the scalability of python as our
code base continues to grow and we are looking at introducing more
products. *I am looking for examples of other people like us (who write
desktop apps in python) with code bases of a similar size who I can
point to (and even better talk to) to help convince him that python is
scalable to 300+ lines of code and beyond. *I have looked at the python
success stories page and haven't come up with anyone quite like us.
One of my project managers questions is: "Are we the only company in the
world with this kind and size of project?"
I want to say no, but am having trouble convincing myself, let alone him.

If you are involved in this kind of thing please get in touch with me.

Perhaps your manager is more concerned about python because he's less
familiar with it, rather than because of scalability problems? Perhaps
using python is actually a competitive advantage for your company?

Have you seen this?
http://www.paulgraham.com/pypar.html

--
Paul Hankin