The major complaint I have about Python is that the packages
which connect it to other software components all seem to have
serious problems. As long as you don't need to talk to anything
outside the Python world, you're fine. But once you do, things
go downhill. MySQLdb has version and platform compatibility
problems. So does M2Crypto. The built-in SSL support is weak.
Even basic sockets don't quite work right; the socket module
encapsulates the timeout mechanism but doesn't get it right.
In the Perl, Java, PHP, and C/C++ worlds, the equivalent
functions just work. That's because, in those worlds, either the
development team for the language or the development team
for the subsystem takes responsibility for making them work.
Only Python doesn't do that.
Python has been around long enough that this should have
been fixed by now.
John Nagle
Jan 24 '07
113  5172 
"George Sakkis" <ge***********@gmail.comwrites:
What does "batteries included" mean to you? To me, it means you don't
have to install add-ons.
So let's make a 500MB executable and add Numpy, Zope, Django, PIL,
pretty much everything actually. Even better, make CheeseShop just a
frontend to a build system that adds and updates automatically
submitted packages to the core. Problem solved ! <wink>.
Numpy should certainly be included and I think there are efforts in
that direction. There is also a movement to choose a web framework to
include and Django might be a good choice. I think the Zope
maintainers want to keep Zope separate and I think PIL has an
incompatible license. I'm not sure what you mean about making
CheeseShop a front end to a build system, but I certainly don't think
random user contributions should get added to the core automatically.
Including a module in the core should carry with it the understanding
that the module has undergone some reasonable evaluation by the core
maintainers and is maintained.
I do think the core should have more stuff than it does, so that its
functionality can be on a par with competing language distros like
J2SE and PHP. Both of those distros include database connectvity
modules and web frameworks. It could be that those other packages can
include more stuff because they have more active developer communities
and can therefore expend more resources maintaining their libraries.
But if that's the case, since the Java and PHP languages themselves
suck compared with Python, we have to ask ourselves why Python has not
been able to attract similar levels of effort and what it could be
doing differently. sk**@pobox.com writes:
database: Sybase, Oracle, MySQL, SQLite, ODBC, Firebird
database versions: who knows? assume two per database
python versions: 2.4, 2.5, 2.6, 3.0?
platforms: windows, pick two linux flavors, solaris
Those numbers give me 6 * 2 * 4 * 4 == 192 combinations to test. Ignore
Python 2.4. That brings it down to 144. Only test on one linux variant and
skip solaris. Now you're at 72. Leave Firebird and Sybase out of the mix.
Now you're down to 48. That's still a fair sized burden.
Why is it that PHP and J2SE manage deal with this problem but Python cannot?
I do think the core should have more stuff than it does, so that its
functionality can be on a par with competing language distros like
J2SE and PHP. Both of those distros include database connectvity
modules and web frameworks.
This is simply not true. J2SE doesn't include a web-framework. J2EE
does, but it's a separate multi-megabyte-download.
PHP _is_ a web centered language, so to say it "includes" a webframework
sounds odd to me.
And _both_ of them don't contain DB connectivity modules! J2SE has JDBC
- which is just a bunch of interfaces. You need to download each and
every driver. Python has DB API, and because it isn't statically typed,
you just download a compliant driver/implementation.
And PHP - well, it hasn't even modules at all. Yes, there are DB
connectors for the important DBs. But for example on debian, you don't
get ORACLE, as they can't ship the needed libs. So you end up compiling
it yourself.
To conclude: given that easy_install stands up to it's promise IMHO, I'm
pretty happy with the batteries included.
Diez
PaulNumpy should certainly be included and I think there are efforts
Paulin that direction. There is also a movement to choose a web
Paulframework to include and Django might be a good choice.
Right off the bat you run into problems. While Numpy is popular in the
scientific computing community it's just dead weight for people without
numerical computing application needs. Besides, the Enthought folks seem to
be doing a pretty good job building a Python distribution geared toward
scientific computing. (Hint: Numpy is just the tip of that particular
iceberg.) The Pylons, web.py, Karrigell, Webware and TurboGears people
might (rightly) feel slighted if you include Django but not their
frameworks.
Where would you stop? At the boundaries of your particular application
interests?
Skip
PaulWhy is it that PHP and J2SE manage deal with this problem but
PaulPython cannot?
I can't speak authoritatively for either PHP or J2SE, but I suspect the
latter at least has some signifiant monetary support (if not outright gobs
of human resources) from Sun. PHP seems to have a more limited application
domain (web apps) from which it has expanded a bit. Can I build a PHP site
out of the box with a PostgreSQL or Oracle backend? Does J2SE have
something comparable to Numpy or scipy?
While might do the occasional bit of Python hacking for free, I still need
to put food on the table. My employer doesn't care if MySQLdb (to use one
of your examples) is delivered with Python or not. They aren't likely to
want to support me to solve your problems.
Skip
Paul Rubin <"http://phr.cx"@NOSPAM.invalidwrites:
Why should the hosting provider need to devote attention to
something like that? MySQLdb or something like it should be
included with Python, not added separately by the hosting provider.
"Something like it" *is* included in Python. Python 2.5 includes
SQLite in the standard library. Where do we draw the line? You want
MySQL, I want PostgreSQL, he wants Firebird, they want an interface to
something proprietary.
The standard library is for modules that are *small* and/or *generally
useful*. SQLite meets the former, and to some extent the latter, far
more than does a MySQL interface.
All the *extra* stuff is what you're paying the hosting company to
take care of in the first place.
--
\ "It's a small world, but I wouldn't want to have to paint it." |
`\ -- Steven Wright |
_o__) |
Ben Finney
Ben Finney <bi****************@benfinney.id.auwrites:
"Something like it" *is* included in Python. Python 2.5 includes
SQLite in the standard library. Where do we draw the line? You want
MySQL, I want PostgreSQL, he wants Firebird, they want an interface to
something proprietary.
Since Python is being touted as good for web apps as a competitor to
PHP, it should offer the same db connectivity in its stdlib that PHP
offers in its. I think that includes MySQL, PostgreSQL, and Oracle,
but I dunno about Firebird.
All the *extra* stuff is what you're paying the hosting company to
take care of in the first place.
No. I'm paying the hosting company for access to a computer that's
connected to electricity and to the internet and which has a
straightforward OS, language package, web server, and db installed.
They need to know how to download a distro and type "make" to build
and install it and occasionally upgrade to a new version, but they
aren't expected to be experts on the intracacies of the software any
more than the guy who sells me potatoes at the supermarket is supposed
to be a gourmet chef. They shouldn't have to deal with dozens of
interdependent modules downloaded from different places just to
support one language. The fewer different packages they have to deal
with, the better off everyone involved is. And that means that a
package like Python should come preconfigured with all the modules
that that a typical user would expect to want to use with it.
Paul Rubin <"http://phr.cx"@NOSPAM.invalidwrites:
Since Python is being touted as good for web apps as a competitor to
PHP
Python is being touted as a good language for *many* purposes, not
just web applications. Python is also a "competitor" to Java, to Ruby,
to Perl, to many other languages. They all have strengths and
weaknesses.
it should offer the same db connectivity in its stdlib that PHP
offers in its.
That doesn't follow at all. Many consider the humungous function
library of PHP to be a significant downside of the system, which
criterion leaves Python ahead. More is not necessarily better.
I'm paying the hosting company for access to a computer that's
connected to electricity and to the internet and which has a
straightforward OS, language package, web server, and db installed.
In which case, there should be no problem with *you* installing
whatever software you need to use the system for what you want.
They shouldn't have to deal with dozens of interdependent modules
downloaded from different places just to support one language.
Either they are providing far more than the minimal set you describe
above, or this is entirely outside their domain. Make up your mind.
You can't claim both that the hosting company should have to maintain
a comprehensive set of functionality, *and* that they should not have
to.
--
\ "Don't worry about what anybody else is going to do. The best |
`\ way to predict the future is to invent it." -- Alan Kay |
_o__) |
Ben Finney
Paul Rubin wrote:
"George Sakkis" <ge***********@gmail.comwrites:
What does "batteries included" mean to you? To me, it means you don't
have to install add-ons.
So let's make a 500MB executable and add Numpy, Zope, Django, PIL,
pretty much everything actually. Even better, make CheeseShop just a
frontend to a build system that adds and updates automatically
submitted packages to the core. Problem solved ! <wink>.
Numpy should certainly be included and I think there are efforts in
that direction. There is also a movement to choose a web framework to
include and Django might be a good choice. I think the Zope
maintainers want to keep Zope separate and I think PIL has an
incompatible license...
do not do that
(1)
i love when i can create a minimalistic system
think about it this way: what if you want to run python on an embeded/
low resource system?
if you want python to do webhosting the solution is _not_ to include
every related package
look at eg. debian: you can use it for lowresource system, desktop,
scientific computation and for webserver as well because of it's
package management system --you can build a min. system and a huge
system as well.
(2)
seriously, python is a programming language and not a flee market (so
don't compare it to java or php)
unfortunately lots of ppl working on web related stuff think web is
the only reason a programming language should exist, which is pretty
stupid
i don't want a "webmodule" in a stdlib at all. implementing the
standards and recommendations should be enough. web in general is a
huge and ugly bloat, keep it away from a language core.
(3)
having a maintained set of modules for every possible problem is nice,
but shouldn't be a part of the core lib.
eg. numpy, mysql, ssl, pil, ... are not needed in the stdlib since
most of the programming tasks don't need those
they should be maintained separately, with an easy way to find and
install them. that's what cheese shop and distutils are for.
for me batteries included means i get a clean and consistent stdlib
and if i need special functionality i can add modules and extensions
easily
nsz
Ben Finney <bi****************@benfinney.id.auwrites:
Since Python is being touted as good for web apps as a competitor to
PHP
Python is being touted as a good language for *many* purposes, not
just web applications. Python is also a "competitor" to Java, to Ruby,
to Perl, to many other languages. They all have strengths and
weaknesses.
Yes but in the cases where Python's weakness compared with one of
those other languages is lack of library functionality, if Python can
remedy the weakness by incorporating similar functionality into its
library it should do so.
That doesn't follow at all. Many consider the humungous function
library of PHP to be a significant downside of the system, which
criterion leaves Python ahead. More is not necessarily better.
I've never heard that as a PHP criticism, at least in any large scale
deployment, which is what a hosting service is.
I'm paying the hosting company for access to a computer that's
connected to electricity and to the internet and which has a
straightforward OS, language package, web server, and db installed.
In which case, there should be no problem with *you* installing
whatever software you need to use the system for what you want.
No. That would be colo or something similar , where I'm basically
paying for bare metal plus electricity and network, and I'm completely
in charge of the software. Web hosting means the ISP is in charge of
all the software except for my application (i.e. they handle the OS,
language package, web server, and db, as described above). So they
run (typically) Linux, MySQL, Apache, and PHP; and I get to upload my
own PHP apps and use the PHP library. That's a lot less work for me
since I don't have to stay on top of kernel patches or firewall
configuration, and it's cheaper because they host a bazillion sites
(virtual hosts) in a a single server instance.
They shouldn't have to deal with dozens of interdependent modules
downloaded from different places just to support one language.
Either they are providing far more than the minimal set you describe
above, or this is entirely outside their domain. Make up your mind.
No it's you who's got it wrong, I just described above what they're
doing. Do you actually use any services like this?
You can't claim both that the hosting company should have to maintain
a comprehensive set of functionality, *and* that they should not have to.
They should get a distro that includes a lot of stuff, type "make",
and all the stuff becomes available to their users.
"Diez B. Roggisch" <de***@nospam.web.dewrites:
This is simply not true. J2SE doesn't include a web-framework. J2EE
does, but it's a separate multi-megabyte-download.
I thought J2SE comes with JSP. Maybe that's not a "framework" in the
fancy sense though.
PHP _is_ a web centered language, so to say it "includes" a
And _both_ of them don't contain DB connectivity modules! J2SE has
JDBC - which is just a bunch of interfaces. You need to download each
and every driver.
Hmm, I thought it came with drivers. Maybe those come with the db
instead? sk**@pobox.com writes:
I can't speak authoritatively for either PHP or J2SE, but I suspect the
latter at least has some signifiant monetary support (if not outright gobs
of human resources) from Sun. PHP seems to have a more limited application
domain (web apps) from which it has expanded a bit. Can I build a PHP site
out of the box with a PostgreSQL or Oracle backend?
I believe so, from having looked at the docs a while back, but I
haven't tried it.
Does J2SE have something comparable to Numpy or scipy?
I don't think so, but Python doesn't either.
While might do the occasional bit of Python hacking for free, I still need
to put food on the table. My employer doesn't care if MySQLdb (to use one
of your examples) is delivered with Python or not. They aren't likely to
want to support me to solve your problems.
The same could be said of Tkinter (a large and complex library module)
or IDLE (a full blown mid-sized application shipped with Python). The
answer is the same for both: if you don't need to use a given module,
then don't. Why would I expect your employer to solve my problems
anyway, even if they relate to some module that you actually use? sk**@pobox.com writes:
iceberg.) The Pylons, web.py, Karrigell, Webware and TurboGears people
might (rightly) feel slighted if you include Django but not their
frameworks.
Yeah well, the Wxpython, PyQt, PyGTK etc. people may feel slighted
that Tkinter got included and their stuff didn't, and the Eric,
Eclipse, Komodo etc. people may feel slighted that IDLE got included,
but that doesn't stop Tkinter and IDLE from being useful and worth
shipping in Python. We were talking about db connectivity modules
which are quite a bit simpler than tkinter. We were also talking
about an SSL wrapper, which *is* included with Python, but is broken
(doesn't examine certificates) so people use external modules instead,
which is just lame.
Where would you stop? At the boundaries of your particular application
interests?
Basic competitive analysis. People ask here all the time "I'm trying
to write application XYZ, should I use language L or should I use
Python" (L is usually Java or PHP but can be other things). There's
always immediately a flood of responses about why Python is better
than language L for application XYZ. If the Pythonistas are serious
about such a claim, competitive analysis says they should be willing
to look at what language L does to support application XYZ (example:
PHP includes database connectivity), make a checklist of L's features,
and see to it that Python achieves (at least) parity in those areas.
In article <7x************@ruckus.brouhaha.com>,
Paul Rubin <http://ph****@NOSPAM.invalidwrote:
I do think the core should have more stuff than it does, so that its
functionality can be on a par with competing language distros like
J2SE and PHP. Both of those distros include database connectvity
modules and web frameworks.
Can't speak for PHP, but J2SE requires the additional installation
of the Mysql Connector/J JDBC driver. Java also does not include a
web framework as far as I can tell, with many people bolting on
struts and Tomcat for this purpose.
Now granted, it may or may not be a good idea for python to copy the
java way of doing things by including the generic database API into
the core libraries rather than expect database module developers to
use the API. But that doesn't solve the core problem that you can't
get database connectivity just through the core language.
And for that matter, perl doesn't include the database drivers or a
web framework either.
php does, but I've always considered php to be a web framework with
an embedded programming language. Comparing python to php here
strikes me as comparing apples and oranges.
Paul Rubin schrieb:
"Diez B. Roggisch" <de***@nospam.web.dewrites:
>This is simply not true. J2SE doesn't include a web-framework. J2EE
does, but it's a separate multi-megabyte-download.
I thought J2SE comes with JSP. Maybe that's not a "framework" in the
fancy sense though.
JSP is also part of J2EE. http://java.sun.com/products/jsp/
"""
Another key concern of the Java EE 5 specification has been the
alignment of its webtier technologies, namely JavaServer Pages (JSP),
JavaServer Faces (JSF), and JavaServer Pages Standard Tag Library (JSTL).
"""
>PHP _is_ a web centered language, so to say it "includes" a
>And _both_ of them don't contain DB connectivity modules! J2SE has
JDBC - which is just a bunch of interfaces. You need to download each
and every driver.
Hmm, I thought it came with drivers. Maybe those come with the db
instead?
Might be, for some. For others where the vendor doesn't care about Java,
they aren't.
And they certainly require special treatment like putting them in the
classpath, setting up the project directory and the like - no simple
import will work out of the box, as it would witch python standard lib
drivers.
diez
Paul Rubin schrieb:
sk**@pobox.com writes:
>I can't speak authoritatively for either PHP or J2SE, but I suspect the
latter at least has some signifiant monetary support (if not outright gobs
of human resources) from Sun. PHP seems to have a more limited application
domain (web apps) from which it has expanded a bit. Can I build a PHP site
out of the box with a PostgreSQL or Oracle backend?
I believe so, from having looked at the docs a while back, but I
haven't tried it.
The docs cover all the available drivers, as they are hard-coded in the
source. Yet you need to specify inclusion of them at compile-time, and
as I said: some distros don't ship with non-OS-drivers.
>Does J2SE have something comparable to Numpy or scipy?
I don't think so, but Python doesn't either.
>While might do the occasional bit of Python hacking for free, I still need
to put food on the table. My employer doesn't care if MySQLdb (to use one
of your examples) is delivered with Python or not. They aren't likely to
want to support me to solve your problems.
The same could be said of Tkinter (a large and complex library module)
or IDLE (a full blown mid-sized application shipped with Python). The
answer is the same for both: if you don't need to use a given module,
then don't. Why would I expect your employer to solve my problems
anyway, even if they relate to some module that you actually use?
As far as I can tell, primary concerns for inclusion of modules are twofold:
- dependencies, which certainly are an issue for DB-modules! Or do you
want every python build to need the oracle OCI drivers installed? Plus
headers?
- maintainer commitment.
Diez
In article <7x************@ruckus.brouhaha.com>,
Paul Rubin <http://ph****@NOSPAM.invalidwrote: sk**@pobox.com writes:
Where would you stop? At the boundaries of your particular application
interests?
If the Pythonistas are serious
about such a claim, competitive analysis says they should be willing
to look at what language L does to support application XYZ (example:
PHP includes database connectivity), make a checklist of L's features,
and see to it that Python achieves (at least) parity in those areas.
Well, digging into this, php includes database connectivity if
compiled against the database client libraries. So it's not as if
you can just download php and have it work against the database you
want.
I also don't think that python as a general-purpose programming
language is in direct competition to web frameworks like PHP and
ColdFusion. Perhaps the right answer is to say that PHP is a better
solution for people wanting a web templating framework with embedded
scripting, while Python is a better solution for people who want a
general scripting/application language with broad support for
specific applications via add-on modules.
But then again, I'm one of those heterodox people who argue that one
should fit the tool to the problem. Common Lisp, bash, awk, sed and
R are other great languages for different domains.
"Diez B. Roggisch" <de***@nospam.web.dewrites:
And they certainly require special treatment like putting them in the
classpath, setting up the project directory and the like - no simple
import will work out of the box, as it would witch python standard lib
drivers.
Well, that's nowhere near as big a deal as having yet another set of
vendors or maintainer to deal with. Minimizing configuration is nice,
but getting your software from as few different sources as possible is
also a big win.
Paul Rubin <http://ph****@NOSPAM.invalidwrites:
"Diez B. Roggisch" <de***@nospam.web.dewrites:
>And they certainly require special treatment like putting them in the
classpath, setting up the project directory and the like - no simple
import will work out of the box, as it would witch python standard lib
drivers.
Well, that's nowhere near as big a deal as having yet another set of
vendors or maintainer to deal with. Minimizing configuration is nice,
but getting your software from as few different sources as possible is
also a big win.
So we should get a better egg support. Then it would all be just a matter of
easy_install <something>. As it is that easy for thousands of modules on CPAN
for Perl.
--
Jorge Godoy <jg****@gmail.com>
Szabolcsseriously, python is a programming language and not a flee
Szabolcsmarket (so don't compare it to java or php)
If it grew to be as bloated as Java it might well become a "flee"
market. ;-)
Skip
PaulWhy would I expect your employer to solve my problems anyway, even
Paulif they relate to some module that you actually use?
Your reasoning seems to be that Python should contain the functional union
of everything at least in Java and PHP if not Perl, Ruby and Tcl as well.
Free software or not, each person who contributes to the development of the
Python core has to put food on the table. As an example, I suspect most
core Java development is funded directly by Sun who pays its employees to
develop libraries, JIT compilers and the like. While my employer pays me to
program in Python I'm not paid to develop core Python code. What little I
do is done on my own time.
If you want to turn the Python distribution into a kitchen sink, make the
argument on python-dev and be prepared to shoulder your share of the burden
should your arguments sway the group as a whole.
Skip
PaulYeah well, the Wxpython, PyQt, PyGTK etc. people may feel slighted
Paulthat Tkinter got included and their stuff didn't, and the Eric,
PaulEclipse, Komodo etc. people may feel slighted that IDLE got
Paulincluded, but that doesn't stop Tkinter and IDLE from being useful
Pauland worth shipping in Python.
They were both pretty much the only games in town when they were included
with Python. If they were developed in today's environment I doubt they
would be included.
PaulBasic competitive analysis. People ask here all the time "I'm
Paultrying to write application XYZ, should I use language L or should
PaulI use Python" (L is usually Java or PHP but can be other things).
So every time some lame ass PHP refugee gripes about something they think
Python is missing which PHP includes we should scan the horizon for all the
candidates and toss them in the next release?
Skip
JorgeSo we should get a better egg support. Then it would all be just
Jorgea matter of easy_install <something>.
I'm all in favor of that...
Skip
Paul Rubin:
I thought J2SE comes with JSP. Maybe that's not a "framework" in the
fancy sense though.
J2SE does not include JSP.
Neil
Chris Mellon wrote:
On 02 Feb 2007 11:10:04 -0800, Paul Rubin
<"http://phr.cx"@nospam.invalidwrote:
"Paul Boddie" <pa**@boddie.org.ukwrites:
If the hosting provider doesn't want to install MySQLdb then it may
not be a technical issue - perhaps they just can't be bothered to
install it, possibly because there's no demand or benefit to the
bottom line in doing so.
Why should the hosting provider need to devote attention to something
like that?
I agree with the remark about paying them for the service. The only
reason why a provider would have Python available, an allocation of
MySQL database instances per user, but not MySQLdb installed, would be
that they want to stick with a conservative set of packages which
requires zero admin (and thus zero outlay on administration), and that
they don't understand obvious requirements for using MySQL with
Python. I don't think they'd get my business.
[...]
There are a number of languages which are primarily used for "web
development". PHP is the *only* one that ships with MySQL client
access.
Ruby doesn't (only primarily web development because of rails)
ASP, either .NET or classic, doesn't.
Java (in any form I'm aware of) doesn't.
You need to get the MySQL Connector/J driver, or whatever it's called
this month.
Cold Fusion doesn't.
Perl doesn't.
Who wants to host at a company that can't install packages anyway?
Quite. I imagine that most GNU/Linux distributions (and various BSDs)
provide at least some version of MySQLdb as a package. If a company
can't manage to provide Python plus MySQL, and then let their users
combine the two by installing a stock package (a single command or
some mouse clicks in the package manager), without offering up
"concerns" about how "secure" such a package might be (which I imagine
some providers might do if they want to discourage people from using
it), then I'd be a bit more concerned about how well they're keeping
up with security updates and how good they are at performing other
elementary administration tasks.
Paul
Paul Rubin wrote:
Ben Finney <bi****************@benfinney.id.auwrites:
Since Python is being touted as good for web apps as a competitor to
PHP
Python is being touted as a good language for *many* purposes, not
just web applications. Python is also a "competitor" to Java, to Ruby,
to Perl, to many other languages. They all have strengths and
weaknesses.
Yes but in the cases where Python's weakness compared with one of
those other languages is lack of library functionality, if Python can
remedy the weakness by incorporating similar functionality into its
library it should do so.
Python should only incorporate functionality in order to offer a
coherent experience (where the omission of functionality would
otherwise lead to a flawed experience). For example, having support
for SSL in the socket module offers a coherent experience because it
means that urllib and related modules can offer to support SSL-related
URLs out of the box. That said, there's probably a case for better
library design and having things like SSL-related "protocol handlers"
be available as plugins, detected and integrated in some kind of
framework, although this would bring in configuration issues of its
own.
[...]
I'm paying the hosting company for access to a computer that's
connected to electricity and to the internet and which has a
straightforward OS, language package, web server, and db installed.
In which case, there should be no problem with *you* installing
whatever software you need to use the system for what you want.
No. That would be colo or something similar , where I'm basically
paying for bare metal plus electricity and network, and I'm completely
in charge of the software.
Or a virtual private server.
Web hosting means the ISP is in charge of all the software except for my application (i.e. they handle the OS,
language package, web server, and db, as described above). So they
run (typically) Linux, MySQL, Apache, and PHP; and I get to upload my
own PHP apps and use the PHP library. That's a lot less work for me
since I don't have to stay on top of kernel patches or firewall
configuration, and it's cheaper because they host a bazillion sites
(virtual hosts) in a a single server instance.
That's known as "shared hosting" these days, I believe. The advantage
of shared hosting is arguably convenience for everyone concerned: the
provider has a static configuration; the users have the stability of a
managed system. However, this only works for people whose needs are
met with that configuration, obviously. The big thing for people
wanting decent Python support in shared hosting environments (aside
from complaints about process isolation) is whether it's easy or
obvious enough for providers to make configurations that Python people
would actually use.
Probably the biggest inhibitor, as far as I can see, has been the
server technology chosen. Many hosting providers have historically
offered no better than CGI for Python, whilst PHP runs within Apache
itself, and it has previously been stated that mod_python has been
undesirable with regard to isolating processes from each other.
Consequently, a number of Python people seem to have held out for
other "high performance" solutions, which various companies now offer.
They shouldn't have to deal with dozens of interdependent modules
downloaded from different places just to support one language.
Either they are providing far more than the minimal set you describe
above, or this is entirely outside their domain. Make up your mind.
No it's you who's got it wrong, I just described above what they're
doing. Do you actually use any services like this?
If a hosting provider claims Python and MySQL support, then I'd hope
that they have worked out that the MySQLdb package glues the two
together. Perhaps better information is necessary for those hosting
companies who haven't worked such things out: you'd have a metapackage
for stuff like this in certain distributions.
You can't claim both that the hosting company should have to maintain
a comprehensive set of functionality, *and* that they should not have to.
They should get a distro that includes a lot of stuff, type "make",
and all the stuff becomes available to their users.
So, for the less forward-thinking providers a metapackage would be the
solution, then?
Paul sk**@pobox.com writes:
PaulYeah well, the Wxpython, PyQt, PyGTK etc. people may feel slighted
Paulthat Tkinter got included and their stuff didn't, and the Eric,
PaulEclipse, Komodo etc. people may feel slighted that IDLE got
Paulincluded, but that doesn't stop Tkinter and IDLE from being useful
Pauland worth shipping in Python.
They were both pretty much the only games in town when they were included
with Python. If they were developed in today's environment I doubt they
would be included.
That would diminish Python's popularity since being able to write GUI
apps without having to download additional crap is a Python selling
point, and IDLE is a considerably more pleasant Python editor than
Notepad is.
PaulBasic competitive analysis. People ask here all the time "I'm
Paultrying to write application XYZ, should I use language L or should
PaulI use Python" (L is usually Java or PHP but can be other things).
So every time some lame ass PHP refugee gripes about something they think
Python is missing which PHP includes we should scan the horizon for all the
candidates and toss them in the next release?
No, just the functions that are requested frequently, like database
adapters.
Dennis Lee Bieber <wl*****@ix.netcom.comwrites:
languages do. PHP hosting providers don't have to install a separate
PHP to MySQL interface gizmo as far as I know.
Really? Then why does the MySQL AB download site at
http://dev.mysql.com/downloads/connector/php/ list both a
"Connector/PHP" AND a "native MySQL driver" for PHP? Along with ODBC,
.NET, and two variations of Java connectors.
Beats me. It does look like PHP5 has stopped bundling the MySQL
client library for licensing reasons: http://www.php.net/manual/en/faq.dat...ses.mysql.php5
It does say "there will always be MySQL support in PHP of one kind or
another" but I'm not sure what that means in this context.
On the other hand if MySQL itself supplies the client library, that's
almost as good. Remember that this isn't purely about minimizing the
number of downloads. It's also about minimizing the number of places
to download from, i.e. the number of different development entities
one has to deal with.
So if you're using Python with MySQL, and you can get everything you
need from python.org and mysql.com, then even if it takes multiple
downloads it's better than having to get additional stuff from random
third party sites, especially when Python's own docs haven't said
where to get the stuff.
Therefore, if someone can persuade mysql.com to offer downloadable
MySQL client drivers for Python at the same place where they supply
the PHP drivers, that's better than the current situation where the
drivers come from some totally unrelated party.
"Paul Boddie" <pa**@boddie.org.ukwrites:
Python should only incorporate functionality in order to offer a
coherent experience (where the omission of functionality would
otherwise lead to a flawed experience). For example, having support
for SSL in the socket module offers a coherent experience because it
means that urllib and related modules can offer to support SSL-related
URLs out of the box.
But they can't, because the built-in socket module SSL interface
doesn't check certificates, causing total security failure if someone
spoofs the remote site. The built-in SSL functionality is broken and
users have to resort to external packages.
Then you have to ask why the stdlib includes anything like urllib in
the first place, under this "coherent experience" concept (I interpret
that as some kind of minimalist philosophy). Can't users have a
coherent experience if the stdlib doesn't include urllib? My own
answer is the one that I thought that the developers had settled on
years ago, namely "batteries included", i.e. ship a rich set of
libraries that provide a wide variety of useful functions, i.e. the
doctrine of minimalism has been explicitly rejected. We then get the
question of whether to include any specific function and that's where
comparisons with other languages come in.
No. That would be colo or something similar ,
Or a virtual private server.
Sure, that would count as "something similar".
If a hosting provider claims Python and MySQL support, then I'd hope
that they have worked out that the MySQLdb package glues the two
together.
I'd expect so too. The issue is there aren't very many of those
companies and that appears partly because of the hassle involved. So
minimizing the hassle suggests itself as a road to wider acceptance.
Your point that shared hosting with Python isn't so easy because of
insufficient isolation between apps is valid. Maybe Python 3.0 can do
something about that and it seems like a valid thing to consider while
fleshing out the 3.0 design.
Perhaps better information is necessary for those hosting
companies who haven't worked such things out: you'd have a metapackage
for stuff like this in certain distributions.
That could help.
So, for the less forward-thinking providers a metapackage would be the
solution, then?
I'm not sure what you mean by metapackage but in general the goal is
to minimize the number of places that the hosting provider (or OS
distro maintainer, or whatever) sk**@pobox.com writes:
PaulWhy would I expect your employer to solve my problems anyway, even
Paulif they relate to some module that you actually use?
Your reasoning seems to be that Python should contain the functional union
of everything at least in Java and PHP if not Perl, Ruby and Tcl as well.
I wouldn't go quite that far. I think there are specific application
areas, such as web server apps, where the Python advocates here on
clpy pitch Python relentlessly against those other languages. Given
that context, Python's stdlib should try to match the libraries of
those other languages in those areas. There are other areas where
Python doesn't get pitched as hard and the other languages have
acknowledged advantages. So it's ok if Python's stdlib gets less
attention in those areas.
Free software or not, each person who contributes to the development
of the Python core has to put food on the table. As an example, I
suspect most core Java development is funded directly by Sun who
pays its employees to develop libraries, JIT compilers and the like.
While my employer pays me to program in Python I'm not paid to
develop core Python code. What little I do is done on my own time.
Again I don't understand what your employer or your activities have to
do with any of this. If you don't want to spend your time developing
Python then don't. You didn't write the interpreter or the tkinter
library or the itertools library and yet there they are. There's
ongoing development that will continue with or without you or me, and
for that matter with or without funding from Sun.
If you want to turn the Python distribution into a kitchen sink,
make the argument on python-dev and be prepared to shoulder your
share of the burden should your arguments sway the group as a whole.
We've had this conversation before and I continue to think your
reasoning above is invalid. I'm not a Python developer, I'm just a
user, and my volunteer coding priorities are elsewhere, as I've
explained before. Python's developers and advocates have a declared
goal of reaching as many users as they can, and as a user I don't mind
offering suggestions about how to do that, but my responsibilities
don't go any further.
Paul Rubin schrieb:
"Diez B. Roggisch" <de***@nospam.web.dewrites:
>And they certainly require special treatment like putting them in the
classpath, setting up the project directory and the like - no simple
import will work out of the box, as it would witch python standard lib
drivers.
Well, that's nowhere near as big a deal as having yet another set of
vendors or maintainer to deal with. Minimizing configuration is nice,
but getting your software from as few different sources as possible is
also a big win.
Ehm... .it _is_ another set of vendors and maintainers. It's just that
they are standard jdbc-compliant. As the python modules are DB API 2.0
compliant - hopefully.
A difference might be that most (not all) of the jdbc-drivers are
java-only, and not have any c-dependencies. Well - nice to have, but
simple inclusion of existing drivers doesn't remove that. Only a
complete rewrite in python would.
Diez
In article <11**********************@m58g2000cwm.googlegroups .com>,
"Paul Boddie" <pa**@boddie.org.ukwrote:
Quite. I imagine that most GNU/Linux distributions (and various BSDs)
provide at least some version of MySQLdb as a package.
Bingo, I've rarely installed python from python.org, or other
libraries from sourceforge, etc., etc.. Usually I've installed
BSD-style ports, debian-style packages, or RedHat-style RPMs.
>
Paul
Kirk Sluder wrote:
In article <11**********************@m58g2000cwm.googlegroups .com>,
"Paul Boddie" <pa**@boddie.org.ukwrote:
Quite. I imagine that most GNU/Linux distributions (and various BSDs)
provide at least some version of MySQLdb as a package.
Bingo, I've rarely installed python from python.org, or other
libraries from sourceforge, etc., etc.. Usually I've installed
BSD-style ports, debian-style packages, or RedHat-style RPMs.
And while Python eggs may be useful for people managing additional
software as some unprivileged user, hosting providers (and virtual
private server administrators) will want packages that fit in with the
rest of the software being managed in the hosting environment.
Paul
Paul Rubin wrote:
"Paul Boddie" <pa**@boddie.org.ukwrites:
Python should only incorporate functionality in order to offer a
coherent experience (where the omission of functionality would
otherwise lead to a flawed experience). For example, having support
for SSL in the socket module offers a coherent experience because it
means that urllib and related modules can offer to support SSL-related
URLs out of the box.
But they can't, because the built-in socket module SSL interface
doesn't check certificates, causing total security failure if someone
spoofs the remote site. The built-in SSL functionality is broken and
users have to resort to external packages.
I was really advocating improvements to the built-in SSL support,
anyway, which was also what the complainant was suggesting before
people started asking him mistakenly why he thought that Python was
weakened by some third party packages (PyOpenSSL, M2Crypto). The
choice here involves either improving the built-in support or
unbundling SSL-based communications altogether. The former option
obviously demands a certain amount of engineering, and then one might
ask why there isn't a convenient framework for plugging in other
flavours of sockets, for example, although there arguably aren't any
as generally important as secure sockets. The latter option needs
everyone to think about how you'd plug such stuff back into Python in
a nice enough way, and then to get people to work on the right
projects to provide something which does the job.
Then you have to ask why the stdlib includes anything like urllib in
the first place, under this "coherent experience" concept (I interpret
that as some kind of minimalist philosophy). Can't users have a
coherent experience if the stdlib doesn't include urllib? My own
answer is the one that I thought that the developers had settled on
years ago, namely "batteries included", i.e. ship a rich set of
libraries that provide a wide variety of useful functions, i.e. the
doctrine of minimalism has been explicitly rejected.
Really, we have to ask whether including the batteries would save
people a lot of work, not just in whether the end-user has to find out
about an external package and then download something, but whether the
logistics around developing the code, integrating it with Python, and
maintaining it would be easier if people just included the stuff with
Python. Here, you need some kind of consensus that feature X needs
supporting and there's an approved way of supporting it which a group
of people would be happy to maintain. Would it benefit the Python
community (including core developers) more if Python shipped with SSL
support out of the box, and would the cost of doing so ultimately be
less than just pointing people at third party libraries and dealing
with their problems?
We then get the question of whether to include any specific function and that's where
comparisons with other languages come in.
Would it benefit the Python community more if Python shipped with
MySQL support out of the box? Is it likely that a user suddenly finds
him/herself needing to connect to a MySQL database? Is it more likely
that the user might suddenly find him/herself needing to download from
a secure site, particularly if some tool (eg. setuptools) suddenly
stumbles across an https URL. Some database systems have a choice of
drivers/libraries/modules (PostgreSQL has quite a few, for example):
choosing a module for standard library inclusion and integrating the
development can be too high a barrier for such questions of inclusion
to be resolved trivially.
[...]
So, for the less forward-thinking providers a metapackage would be the
solution, then?
I'm not sure what you mean by metapackage but in general the goal is
to minimize the number of places that the hosting provider (or OS
distro maintainer, or whatever)
On various distributions you get packages which don't actually contain
anything, but which indicate a suite of packages which are to be
installed. So, if you install the ubuntu-desktop package on Ubuntu
systems, you get the GNOME desktop environment and all the
dependencies (and a bunch of other stuff). Perhaps there should be a
python-mysql-hosting package for such providers.
Paul
"Paul Boddie" <pa**@boddie.org.ukwrites:
And while Python eggs may be useful for people managing additional
software as some unprivileged user, hosting providers (and virtual
private server administrators) will want packages that fit in with the
rest of the software being managed in the hosting environment.
And why eggs wouldn't satisfy them? Eggs can be installed globally as well,
making the package available to every client of this hosting server (if they
mount their libs from a unique NFS server then it would automatically be
available for all of their servers).
--
Jorge Godoy <jg****@gmail.com>
In article <11*********************@m58g2000cwm.googlegroups. com>,
"Paul Boddie" <pa**@boddie.org.ukwrote:
Would it benefit the Python community more if Python shipped with
MySQL support out of the box? Is it likely that a user suddenly finds
him/herself needing to connect to a MySQL database?
The other problem is that it chains the python release schedule to
that of MySQL AB (and postgresql, and whatever.) One of the key
advantages to independent modules is that I don't need to update my
entire python system every time my database vendor releases a new
library, nor do I have to accept lag time as the module is tested
and rolled into the base distribution. Database client libraries
are much more of a moving target than core language features.
I find it interesting that PHP struggled with these issues and
decided to abandon embedded MySQL support partly because they
couldn't maintain their own parallel versions of the client
libraries. http://us2.php.net/manual/en/faq.dat...bases.mysql.ph
p5
Among the other problems faced by both PHP and Python in bundling
MySQL support is that they can't legally do it without adopting the
GPL.
Which leaves me wondering why the python core should adopt a feature
that was abandoned by PHP, and was never highly recommended or used?
Paul
I find Paul Rubin's arguments compelling and convincing. As just a
Python user (i.e., someone who has contributed nothing) let me add a
few comments along the same lines.
On 03 Feb 2007 18:31:03 -0800, Paul Rubin
<"http://phr.cx"@nospam.invalidwrote:
sk**@pobox.com writes:
PaulWhy would I expect your employer to solve my problems anyway, even
Paulif they relate to some module that you actually use?
Your reasoning seems to be that Python should contain the functional union
of everything at least in Java and PHP if not Perl, Ruby and Tcl as well.
I wouldn't go quite that far. I think there are specific application
areas, such as web server apps, where the Python advocates here on
clpy pitch Python relentlessly against those other languages. Given
that context, Python's stdlib should try to match the libraries of
those other languages in those areas. There are other areas where
Python doesn't get pitched as hard and the other languages have
acknowledged advantages. So it's ok if Python's stdlib gets less
attention in those areas.
In fact, it is quite frustrating to operate under the impression that
"Python is good for X, Y, Z, ...." and then realize "ooops, it is
becoming a pain in the ass to do X, whereas language L does X just
fine". It seems to me that Python advocates sometimes (often?) get
carried away. DB "sure, no problem"; web-frameworks "who needs Rails,
we have (lots of) frameworks that do it"; functional programming
"Python can do all the functional programming anyone in his/her mind
should ever try to do"; etc.
Compare this to the, in my opinion, equanimous, fair, "advertisement"
one finds in the Erlang page or the recognition by schemers that
scheme is not the only game in town.
>
If you want to turn the Python distribution into a kitchen sink,
make the argument on python-dev and be prepared to shoulder your
share of the burden should your arguments sway the group as a whole.
We've had this conversation before and I continue to think your
reasoning above is invalid. I'm not a Python developer, I'm just a
user, and my volunteer coding priorities are elsewhere, as I've
explained before. Python's developers and advocates have a declared
goal of reaching as many users as they can, and as a user I don't mind
offering suggestions about how to do that, but my responsibilities
don't go any further.
R.
--
http://mail.python.org/mailman/listinfo/python-list
--
Ramon Diaz-Uriarte
Statistical Computing Team
Structural Biology and Biocomputing Programme
Spanish National Cancer Centre (CNIO) http://ligarto.org/rdiaz
Jorge Godoy wrote:
"Paul Boddie" <pa**@boddie.org.ukwrites:
And while Python eggs may be useful for people managing additional
software as some unprivileged user, hosting providers (and virtual
private server administrators) will want packages that fit in with the
rest of the software being managed in the hosting environment.
And why eggs wouldn't satisfy them? Eggs can be installed globally as well,
making the package available to every client of this hosting server (if they
mount their libs from a unique NFS server then it would automatically be
available for all of their servers).
Because Python is just another thing to support for various hosting
providers. One of the problems people supposedly have when persuading
such companies to add or update packages is, I imagine, a lack of
familiarity those companies have with Python technologies. Asking them
to use a technology-specific packaging system might be a bit much if
they aren't even familiar with (or interesting in knowing more about)
the packages required to satisfy their customers' basic needs.
Moreover, there's a lot of software in the average GNU/Linux or BSD
distribution, but if you had to "break out" into technology-specific
package/dependency managers to install some of the more heterogeneous
packages, it would be a nightmare: perhaps installing the
documentation would have you visit CPAN for some Perl scripts before
throwing you into the TeX package manager to get some missing TeX
libraries; then you might be off into setuptools for the Python
scripting extensions, possibly via equivalent tools for other
scripting extensions; finally, you'd be off via any other system
thought essential to manage software from a particular parochial
technological viewpoint. I've had to install software like this, and
it takes huge amounts of time for no good reason if you can use system
packages instead.
Paul
Paul Rubin wrote:
"George Sakkis" <ge***********@gmail.comwrites:
>>What does "batteries included" mean to you? To me, it means you don't
have to install add-ons.
So let's make a 500MB executable and add Numpy, Zope, Django, PIL,
pretty much everything actually. Even better, make CheeseShop just a
frontend to a build system that adds and updates automatically
submitted packages to the core. Problem solved ! <wink>.
Numpy should certainly be included and I think there are efforts in
that direction.
As I understand it, the effort is directed towards elaborating the
current array module so that it handles multi-dimensional array.
This would be a good step but numpy os much more and is of minority
interest and so should probably not, in my opinion, be included in the
Python core.
Colin W.
There is also a movement to choose a web framework to
include and Django might be a good choice. I think the Zope
maintainers want to keep Zope separate and I think PIL has an
incompatible license. I'm not sure what you mean about making
CheeseShop a front end to a build system, but I certainly don't think
random user contributions should get added to the core automatically.
Including a module in the core should carry with it the understanding
that the module has undergone some reasonable evaluation by the core
maintainers and is maintained.
I do think the core should have more stuff than it does, so that its
functionality can be on a par with competing language distros like
J2SE and PHP. Both of those distros include database connectvity
modules and web frameworks. It could be that those other packages can
include more stuff because they have more active developer communities
and can therefore expend more resources maintaining their libraries.
But if that's the case, since the Java and PHP languages themselves
suck compared with Python, we have to ask ourselves why Python has not
been able to attract similar levels of effort and what it could be
doing differently.
Paul Boddie wrote:
I was really advocating improvements to the built-in SSL support,
anyway, which was also what the complainant was suggesting before
people started asking him mistakenly why he thought that Python was
weakened by some third party packages (PyOpenSSL, M2Crypto).
The real problems are with integration of modules that
aren't written in Python. Python-only modules don't have the
version compatibility problems which C modules do. Loading
a Python-only module from an external source is usually not
a big deal. Building a C module, especially one with
dependencies on other components, can be a big deal.
So, focus on problems with modules which have C
components.
John Nagle
John Nagle wrote:
>
The real problems are with integration of modules that
aren't written in Python. Python-only modules don't have the
version compatibility problems which C modules do. Loading
a Python-only module from an external source is usually not
a big deal. Building a C module, especially one with
dependencies on other components, can be a big deal.
So, focus on problems with modules which have C
components.
I think that the integration of certain "popular" extensions into the
Python distribution has been one of the tools employed to mitigate the
version compatibility situation: if the popular stuff gets released
with Python, perhaps fewer people will complain about "C modules"
whose developers/maintainers haven't updated them for the latest
release. I'm not really convinced that this makes for a sustainable
approach, however, since this just becomes a process of accretion
where the core development community must get larger and become very
well-coordinated in order to manage the dependencies within the
software, as well as tracking external dependencies.
Over the years there have been suggestions which might have made the
compatibility situation a bit better: a "Python in a tie" release was
an objective of the seemingly dormant Python Business Foundation;
Linux Standard Base inclusion of Python has been proposed and
discussed. I can't help feeling that without improvements in such
other areas, things like "C module" availability for different Python
versions (and for different versions of related libraries) is mainly
an issue of doing the almost thankless hard work of backporting,
testing and managing different build configurations. Still, there may
be demand for a backports community in all this.
Paul
On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
"Paul Boddie" <p...@boddie.org.ukwrites:
>Probably the biggest inhibitor, as far as I can see, has been the
server technology chosen. Many hosting providers have historically
offered no better than CGI for Python, whilst PHP runs within Apache
itself, and it has previously been stated that mod_python has been
undesirable with regard to isolating processes from each other.
Consequently, a number of Python people seem to have held out for
other "high performance" solutions, which various companies now offer.
Your point that shared hosting with Python isn't so easy because of
insufficient isolation between apps is valid. Maybe Python 3.0 can do
something about that and it seems like a valid thing to consider while
fleshing out the 3.0 design.
To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.
First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process. These
distinct sub interpreters can be linked to different parts of the URL
namespace. This means that it is possible to host more than one
mod_python application where each executes within in their own
distinct sub interpreter. The outcome of this is that each application
can have their own sys.path, own os.environ, own sets of modules and
potentially with different versions of some module.
Maintaining separation using sub interpreters eliminates the bulk of
problems with applications interfering which each other at least
within a process. Some problems can still arise though where third
party extension modules for Python aren't written so as to be usable
from multiple sub interpreters at the same time however. This is not a
failing of mod_python though, but a failing of the module writers.
The main area where interference can occur is where applications needs
to write to the file system. This is because all code will be
executing as the user that Apache runs as. Thus, distinct applications
could overwrite each others data within the file system. On one level
this just means that applications need to be configured to always use
their own part of the file system. For example, if using the support
in mod_python for sessions, the distinct applications should perhaps
use separate session databases rather than use the same common
database. Ultimately though, a rogue application could write where
ever it wants to, but from what I know (and could be wrong), this
isn't different to other languages systems within Apache such as PHP
and mod_perl.
Another possibility for interference is where an application simply
does something bad like get stuck in a tight loop or consume lots of
memory. Such an event can possibly interfere with other applications,
even across language boundaries, however, how bad the impact will be
depend on what MPM is used by Apache.
If "prefork" MPM is used, then the request being handled by that
application is the only thing which would be running within that child
process at that particular time. Thus, if it stops the functioning of
just that one process it doesn't matter as Apache will just farm
requests off to other child processes. If that initial child process
crashes because of the problem, then again it doesn't matter as Apache
will just create another child process to replace it and will
otherwise keep running.
If the "worker" MPM is used the impact can be greater as there could
be other requests being handled concurrently within the same child
process and the performance of those requests may be hindered. If the
worst happens and the child process crashes, only other requests being
handled within that specific child process would be affected, those in
other child processes would again continue unaffected as would Apache
as a whole.
The worst case is the "winnt" MPM (ie., Windows boxes). This is
because there is only one Apache process and thus a rogue application
can affect the whole Apache web server.
It should be highlighted though that this class of problem is not
unique to Python or mod_python as you could get rogue code in a PHP
page or mod_perl application just as easily.
What it all really comes down to is that the only features that are
really missing are the ability for distinct applications to run as
distinct users, or for applications to run inside of some sort of
chroot environment. Some aspects of this are addressed by FCGI and
SCGI, but again lack of this feature within mod_python itself is not
unique to it and that as far as I know is also going to be an issue
for other language systems for Apache such as PHP or mod_perl.
Having said all that, perhaps those who are complaining about lack of
support for specific features in mod_python can now clarify what
actually you are talking about. At the moment the brief comments being
made seem to possibly cover some things that mod_python can already do
but may not be obvious.
Graham
Graham Dumpleton wrote:
>
Having said all that, perhaps those who are complaining about lack of
support for specific features in mod_python can now clarify what
actually you are talking about. At the moment the brief comments being
made seem to possibly cover some things that mod_python can already do
but may not be obvious.
I had a more complete response to this before Google Groups and my
browser went into "stupid mode" together, but I'd like to say that I
don't have any complaints about mod_python, but I know that there were
people who claimed [1] that PHP had its "safe mode" [2] which gave
isolation to applications belonging to different users in the same
Apache instance, asserting that mod_python not having that particular
feature prevented wider adoption of Python in the hosting business.
It's good to see a clarification of mod_python and its position in
relation to such issues, though. I don't really share the concerns
mentioned, and wouldn't want to be using or providing services of the
kind previously discussed, anyway. Instead, I'd want to deploy
applications using other techniques such as virtualisation, rather
than throwing customers into the same Web server and, apparently in
the case of PHP, relying on hearsay about whether they can interfere
with each other. I note that the creator of mod_python seems to have a
similar perspective on virtualisation, given the nature of his hosting
company's services.
Paul
[1] http://groups.google.com/group/comp.lang.python/msg/
469dd47f5c1ad521
[2] http://no.php.net/features.safe-mode
Graham Dumpleton wrote:
On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
>>"Paul Boddie" <p...@boddie.org.ukwrites:
>>>Probably the biggest inhibitor, as far as I can see, has been the
server technology chosen. Many hosting providers have historically
offered no better than CGI for Python, whilst PHP runs within Apache
itself, and it has previously been stated that mod_python has been
undesirable with regard to isolating processes from each other.
Consequently, a number of Python people seem to have held out for
other "high performance" solutions, which various companies now offer.
Your point that shared hosting with Python isn't so easy because of
insufficient isolation between apps is valid. Maybe Python 3.0 can do
something about that and it seems like a valid thing to consider while
fleshing out the 3.0 design.
To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.
First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process.
Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.
FastCGI, though, can get all the CPUs going. It takes more
memory, though, since each instance has a full copy of Python
and all the libraries in use.
(FastCGI is a straightforward transaction processing engine.
Each transaction program is launched in a separate process, and,
once done with one transaction, can be used to do another one
without reloading. When things are slow, the extra transaction processes
are told to exit; when load picks up, more of them are forked.
Security is comparable to CGI.)
John Nagle
On Feb 5, 9:45 am, John Nagle <n...@animats.comwrote:
Graham Dumpleton wrote:
On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
>"Paul Boddie" <p...@boddie.org.ukwrites:
>>Probably the biggest inhibitor, as far as I can see, has been the
server technology chosen. Many hosting providers have historically
offered no better than CGI for Python, whilst PHP runs within Apache
itself, and it has previously been stated that mod_python has been
undesirable with regard to isolating processes from each other.
Consequently, a number of Python people seem to have held out for
other "high performance" solutions, which various companies now offer.
>Your point that shared hosting with Python isn't so easy because of
insufficient isolation between apps is valid. Maybe Python 3.0 can do
something about that and it seems like a valid thing to consider while
fleshing out the 3.0 design.
To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.
First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process.
Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.
That is not true if 'prefork' MPM is used for Apache which is how most
people seem to run it. This is because each Apache child process only
run one request at a time and so there isn't normally going to be any
contention on the GIL at all. The only case where there would be
contention in this arrangement is if the request handlers within
Apache had spawned off distinct threads themselves to do stuff. Even
then, in this arrangement the main request handler is usually not
doing anything and is just waiting for the created thread to finish
what it was doing. Thus if only one thread was spawned to do some work
or a blocking operation to allow the main thread to timeout, then
again there isn't really any contention as only one thread is actually
doing anything. If you are trying to embed very intensive operations
with threads within Apache then I would suggest it is not really the
best design you could use anyway as such things would be much better
farmed off to a long running backend process using XML-RPC or some
other interprocess communication mechanism.
If one is using the "worker" MPM then yes there will be some
contention if multiple requests are being handled by mod_python at the
same time within the same Apache child process. The downside of this
is lessened however by the fact that there are still multiple Apache
child processes and Apache will spread requests across all the Apache
child processes, thus the amount that may be running concurrently
within any one process is less.
The basic problem of GIL contention here is no different to a single
Python backend process which is handling everything behind Apache. In
some respects the Apache approach actually works better as there are
multiple processes spreading the load. Your comment on the GIL is
therefore partly unjustified in that respect for Apache and
mod_python. Your statement in some respect still stands for Python
itself when run as a single process, but you linked it to mod_python
and Apache which lessens the impact through its architecture of using
multiple child processes.
Finally we have 'winnt' MPM, again, because this is all in the one
process you will have GIL contention in a much more substantial
manner. However, I'd suggest that most wouldn't choose Apache on
Windows as a major deployment platform.
FastCGI, though, can get all the CPUs going. It takes more
memory, though, since each instance has a full copy of Python
and all the libraries in use.
How is that any different to Apache child processes. Each Apache child
process has a full copy of Python and the libraries in use. Each
Apache child process can be making use of different CPUs. Further,
static file requests, plus other requests against PHP, mod_perl etc
can when mod_python is also running be on separate CPUs within the
same child process when 'worker' MPM is being used. Thus you haven't
lost all forms or parallelism that may be possible, it is only within
the mod_python world that there will be some GIL contention and only
with 'worker' and 'winnt' MPMs, not 'prefork'. It isn't going to lock
out non Python stuff from making use of additional CPUs.
(FastCGI is a straightforward transaction processing engine.
Each transaction program is launched in a separate process, and,
once done with one transaction, can be used to do another one
without reloading. When things are slow, the extra transaction processes
are told to exit; when load picks up, more of them are forked.
Security is comparable to CGI.)
Apache will also kill off excess child processes when it deems they
are no longer required, or create new ones as demand dictates.
So, I am still not sure where the big issue is, the architecture of
Apache limits the impact of GIL contention in ways that Python alone
doesn't.
Graham
John Nagle wrote:
Graham Dumpleton wrote:
On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
>"Paul Boddie" <p...@boddie.org.ukwrites:
Probably the biggest inhibitor, as far as I can see, has been the
server technology chosen. Many hosting providers have historically
offered no better than CGI for Python, whilst PHP runs within Apache
itself, and it has previously been stated that mod_python has been
undesirable with regard to isolating processes from each other.
Consequently, a number of Python people seem to have held out for
other "high performance" solutions, which various companies now offer.
Your point that shared hosting with Python isn't so easy because of
insufficient isolation between apps is valid. Maybe Python 3.0 can do
something about that and it seems like a valid thing to consider while
fleshing out the 3.0 design.
To clarify some points about mod_python, since these posts do not
properly explain the reality of the situation and I feel people are
getting the wrong impression.
First off, when using mod_python it is possible to have it create
multiple sub interpreters within each Apache child process.
Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.
The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting. sj*******@yahoo.com wrote:
John Nagle wrote:
>>Graham Dumpleton wrote:
>>>On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
"Paul Boddie" <p...@boddie.org.ukwrites:
> Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.
The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.
Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request. For
many AJAX apps, the loading cost tends to dominate the transaction.
FastCGI, though, reuses the loaded Python (or whatever) image.
The code has to be written to be able to process transactions
in sequence (i.e. don't rely on variables intitialized at load),
but each process lives for more than one transaction cycle.
However, each process has a copy of the whole Python system,
although maybe some code gets shared.
John Nagle
On 5 Feb, 18:52, John Nagle <n...@animats.comwrote:
>
Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request. For
many AJAX apps, the loading cost tends to dominate the transaction.
According to the Apache prefork documentation, you can configure how
often the child processes stick around... http://httpd.apache.org/docs/2.2/mod...questsperchild
....and I suppose mod_python retains its state in a child process as
long as that process is running:
"Once created, a subinterpreter will be reused for subsequent
requests. It is never destroyed and exists until the Apache process
dies."
- http://www.modpython.org/live/curren...i-interps.html
"Depending on the use of various PythonInter* directives, a single
python interpreter (and list of imported modules, and per-module
global variables, etc) might be shared between multiple mod_python
applications."
- http://www.modpython.org/FAQ/faqw.py...=faq03.005.htp
The FAQ entry (3.1) about module reloading would appear to be
pertinent here, too: http://www.modpython.org/FAQ/faqw.py...=faq03.001.htp
Paul
John Nagle <na***@animats.comwrites:
The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.
Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request. For
many AJAX apps, the loading cost tends to dominate the transaction.
I think the idea is that each pre-forked subprocess has its own
mod_python that services multiple requests serially.
New to me is the idea that you can have multiple separate Python
interpreters in a SINGLE process (mentioned in another post). I'd
thought that being limited to one interpreter per process was a
significant and hard-to-fix limitation of the current CPython
implementation that's unlikely to be fixed earlier than 3.0.
On Feb 6, 4:52 am, John Nagle <n...@animats.comwrote:
sjdevn...@yahoo.com wrote:
John Nagle wrote:
>Graham Dumpleton wrote:
>>On Feb 4, 1:05 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:
>>>"Paul Boddie" <p...@boddie.org.ukwrites:
Realistically, mod_python is a dead end for large servers,
because Python isn't really multi-threaded. The Global Python
Lock means that a multi-core CPU won't help performance.
The GIL doesn't affect seperate processes, and any large server that
cares about stability is going to be running a pre-forking MPM no
matter what language they're supporting.
Pre-forking doesn't reduce load; it just improves responsiveness.
You still pay for loading all the modules on every request. For
many AJAX apps, the loading cost tends to dominate the transaction.
FastCGI, though, reuses the loaded Python (or whatever) image.
The code has to be written to be able to process transactions
in sequence (i.e. don't rely on variables intitialized at load),
but each process lives for more than one transaction cycle.
However, each process has a copy of the whole Python system,
although maybe some code gets shared.
As someone else pointed out, your understanding of how mod_python
works within Apache is somewhat wrong. I'll explain some things a bit
further to make it clearer for you.
When the main Apache process (parent) is started it will load all the
various Apache modules including that for mod_python. Each of these
modules has the opportunity to hook into various configuration phases
to perform actions. In the case of mod_python it will hook into the
post config phase and initialise Python which will in turn setup all
the builtin Python modules.
When Apache forks off child processes each of those child processes
will inherit Python already in an initialised state and also the
initial Python interpreter instance which was created, This therefore
avoids the need to perform initialisation of Python every time that a
child process is created.
In general this initial Python interpreter instance isn't actually
used though, as the default strategy of mod_python is to allocate
distinct Python interpreter instances for each VirtualHost, thereby at
least keeping applications running in distinct VirtualHost containers
to be separate so they don't interfere with each other.
Yes, these per VirtualHost interpreter instances will only be created
on demand in the child process when a request arrives which
necessitates it be created and so there is some first time setup for
that specific interpreter instance at that point, but the main Python
initialisation has already occurred so this is minor. Most
importantly, once that interpreter instance is created for the
specific VirtualHost in the child process it is retained in memory and
used from one request to the next. If the handler for a request loads
in Python modules, those Python modules are retained in memory and do
not have to be reloaded on each request as you believe.
If you are concerned about the fact that you don't specifically know
when an interpreter instance will be first created in the child
process, ie., because it would only be created upon the first request
arriving that actually required it, you can force interpreter
instances to be created as soon as the child process has been created
by using the PythonImport directive. What this directive allows you to
do is specify a Python module that should be preloaded into a specific
interpreter instance as soon as the child process is created. Because
the interpreter will need to exist, it will first be created before
the module is loaded thereby achieving the effect of precreating the
specific named interpreter instance.
So as to make sure you don't think that that first interpreter
instance created in the parent and inherited by the child processes is
completely wasted, it should be pointed out that the first interpreter
instance created by Python is sort of special. In general it shouldn't
matter, but there is one case where it does. This is where a third
party extension module for Python has not been written so as to work
properly in a context where there are multiple sub interpreters.
Specifically, if a third party extension module used the simplified
API for GIL locking one can have problems using that module in
anything but the first interpreter instance created by Python. Thus,
the first instance is retained and in some cases it may be necessary
to force your application to run within the context of that
interpreter instance to get it to work where using such a module. If
you have to do this for multiple applications running under different
VirtualHost containers you loose your separation though, thus this is
only provided as a fallback when you don't have a choice.
I'll mention one other area in case you have the wrong idea about it
as well. In mod_python there is a feature for certain Python modules
to be reloaded. This feature is normally on by default but is always
recommended to be turned off in a production environment. To make it
quite clear, this feature does not mean that the modules which are
candidates for reloading will be reloaded on every request. Such
modules will only be reloaded if the code file for that module has
been changed. Ie., its modification time on disk has been changed. In
mod_python 3.3 where this feature is a bit more thorough and robust,
it will also reload a candidate module if some child or descendant of
the module has been changed.
So to summarise. Interpreter instances once created in the child
processes for a particular context are retained in memory and used
from one request to the next. Further, any modules loaded by code for
a request handler is retained in memory and do not have to be reloaded
on each request. Even when module reloading is enabled in mod_python,
modules are only reloaded where a code file associated with that
module has been changed on disk.
Does that clarify any misunderstandings you have?
So far it looks like the only problem that has been identified is one
that I already know about, which is that there isn't any good
documentation out there which describes how it all works. As a result
there are a lot of people out there who describe wrongly how they
think it works and thus give others the wrong impression. I already
knew this, as I quite often find myself having to correct statements
on various newgroups and in documentation for various Python web
frameworks. What is annoying is that even though you point out to some
of the Python web frameworks that what they state in their
documentation is wrong or misleading they don't correct it. Thus the
wrong information persists and keeps spreading the myth that there
must be some sort of problem where there isn't really. :-(
Graham This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Brandon J. Van Every |
last post by:
What's better about Ruby than Python? I'm sure there's something. What is
it?
This is not a troll. I'm language shopping and I want people's answers. I
don't know beans about Ruby or have...
|
by: Brandon J. Van Every |
last post by:
I'm realizing I didn't frame my question well.
What's ***TOTALLY COMPELLING*** about Ruby over Python? What makes you jump
up in your chair and scream "Wow! Ruby has *that*? That is SO...
|
by: mike420 |
last post by:
I think everyone who used Python will agree that its syntax is
the best thing going for it. It is very readable and easy
for everyone to learn. But, Python does not a have very good
macro...
|
by: Ilias Lazaridis |
last post by:
"
A cooperation between Sun Microsystems and IBM&Co. in conjunction with
liberal & high evolutive communities would result in an nearly
unbeatable programming platform.
My evaluation has shown:...
|
by: Ivan Reborin |
last post by:
Hello everyone,
I was wondering if anyone here has a moment of time to help me with 2
things that have been bugging me.
1. Multi dimensional arrays - how do you load them in python
For...
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: nemocccc |
last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: Hystou |
last post by:
There are some requirements for setting up RAID:
1. The motherboard and BIOS support RAID configuration.
2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers,...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...
| |
