Python Web Programming - looking for examples of solid high-traffic sites

[Victor Kryukov]

Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago.

Today, as we get better idea of what we need, we're going to re-write
everything from scratch. Python is an obvious candidate for our team:
everybody knows it, everybody likes it, it has *real* objects, nice
clean syntax etc.

Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

Our problem is - we yet have to find any example of high-traffic,
scalable web-site written entirely in Python. We know that YouTube is
a suspect, but we don't know what specific python web solution was
used there.

TurboGears, Django and Pylons are all nice, and provides rich features
- probably too many for us - but, as far as we understand, they don't
satisfy the stability requirement - Pylons and Django hasn't even
reached 1.0 version yet. And their provide too thick layer - we want
something 'closer to metal', probably similar to web.py -
unfortunately, web.py doesn't satisfy the stability requirement
either, or so it seems.

So the question is: what is a solid way to serve dynamic web pages in
python? Our initial though was something like python + mod_python +
Apache, but we're told that mod_python is 'scary and doesn't work very
well'.

And although http://www.python.org/about/quotes/ lists many big names
and wonderful examples, be want more details. E.g. our understanding
is that Google uses python mostly for internal web-sites, and
performance is far from perfect their. YouTube is an interesting
example - anybody knows more details about that?

Your suggestions and comments are highly welcome!

Best Regards,
Victor.

[John Nagle]

Jarek Zgoda wrote:

Victor Kryukov napisa³(a):

>>Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

I don't think you find anything even remotely resembling that idea here.
Moreover, I don't think you find it elsewhere. Maybe even such tools do
not exist in nature?

Sure they do. I have a complex web site, "http://www.downside.co m",
that's implemented with Perl, Apache, and MySQL. It automatically reads SEC
filings and parses them to produce financial analyses. It's been
running for seven years, and hasn't been modified in five, except once
when the NASDAQ changed the format of their ticker symbol file.

It's not that useful at this point, because its purpose was to predict
failing dot-coms, but it's still running and doing a sizable amount of
work every day to update itself.

John Nagle

[Alex Martelli]

Victor Kryukov <vi************ @gmail.comwrote :
...

And although http://www.python.org/about/quotes/ lists many big names
and wonderful examples, be want more details. E.g. our understanding
is that Google uses python mostly for internal web-sites, and
performance is far from perfect their. YouTube is an interesting
example - anybody knows more details about that?

Hmmm, I do, but exactly because I _do_ know a lot more details I cannot
discuss them unless you're under the proper Non-Disclosure Agreement
with Google, Inc. The best I can do otherwise is to point you to
already existing webpages -- I'm not going to reveal Google-Confidential
information, nor go to the substantial trouble to get such info cleared
by Legal and all other Google stakeholders.

For example, it HAS been published elsewhere that YouTube uses lighttpd,
not Apache: <http://trac.lighttpd.n et/trac/wiki/PoweredByLightt pd>.

Fortunately, I managed to convince my YouTube colleagues to publically
present many more details about the evolution of their architecture
which had been discussed in Google-confidential talks and presentations
-- and my wife Anna, who's on the program selection committee of OSCON,
may have helped that talk get accepted (must not have been a hard
job:-). See:
<http://conferences.ore illynet.com/cs/os2007/view/e_sess/13435>

I hope to see you (and anybody else interested in solid technical
details about using Python for websites on YouTube's scale) in Portland,
OR on July 26 -- that will also be your first and best chance to ask
Mike Solomon specific questions that his talk might not directly
address. Once that's done, maybe somebody can convince the YouTube guys
to contribute a "Python Success Story" so that future querants about
this can be easily pointed to a URL!-)

I would also encourage anybody who's so keenly interested in Python to
visit our jobs-listing web app, e.g., if within the US, at
<http://www.google.com/support/jobs/b...00&dep_id=1173
&by_loc=1>. Of course, one should *never* have the implementation
language of a web-app show up in the URL, and I believe we've carefully
avoided that error in other external-facing web-apps, such as (one I can
reveal is indeed in Python, because that was mentioned at
<http://www.sauria.com/~twl/conferenc.../Python%20at%2
0Google.html>) code.google.com . Etc, etc.

Performance of web-apps (be they internal or external) depends more on
the architecture than on the implementation language (as long as highly
optimized C or C++, NOT Java or Python or whatever, is used for the few
jobs that are extremely CPU-intensive, such as codecs, of course:-).

So, if some Python-coded internal web-apps at Google perform badly
(which may be the case as you say, though I can't think of any
off-hand), it must be an issue of architecture. For example, a heavily
used internal web-app at Google is Mondrian,
<http://www.niallkenned y.com/blog/archives/2006/11/google-mondrian.html>
, Guido van Rossum's web-app for code review -- it's got a good, solid
architecture, and its performance is so good that many Googlers, me
included, have switched to it for all the reviews we do (and, believe
me, we do MANY -- _nothing_ is submitted to the global code repository
until it's been OK'd in a code review). I can't mention other such apps
because, AFAIK, they haven't been previously talked about in public and
so they're Google Confidential by default until otherwise determined.
Alex

[Daniel Nogradi]

For example, it HAS been published elsewhere that YouTube uses lighttpd,

not Apache: <http://trac.lighttpd.n et/trac/wiki/PoweredByLightt pd>.

How do you explain these, then:

http://www.youtube.com/results.xxx
http://www.youtube.com/results.php
http://www.youtube.com/results.py

Just wondering,
Daniel

[Jarek Zgoda]

Daniel Nogradi napisa³(a):

>For example, it HAS been published elsewhere that YouTube uses lighttpd,
not Apache: <http://trac.lighttpd.n et/trac/wiki/PoweredByLightt pd>.

How do you explain these, then:

http://www.youtube.com/results.xxx
http://www.youtube.com/results.php
http://www.youtube.com/results.py

Server signature is usually configurable.

--
Jarek Zgoda

"We read Knuth so you don't have to."

[Bruno Desthuilliers]

John Nagle a écrit :

Victor Kryukov wrote:

>Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago.

...

>Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

You may not be happy with Python, then.

John, I'm really getting tired of your systemic and totally
unconstructive criticism. If *you* are not happy with Python, by all
means use another language.

[Bruno Desthuilliers]

Istvan Albert a écrit :

On May 16, 5:04 pm, Victor Kryukov <victor.kryu... @gmail.comwrote :

>Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

I think this is a requirement that is pretty much impossible to
satisfy. Only dead frameworks stay the same. I have yet to see a
framework that did not have incompatible versions.

Django has a very large user base, great documentation and is deployed
for several online new and media sites. It is fast, it's efficient and
is simple to use. Few modern frameworks (in any language) are
comparable, and I have yet to see one that is better,

Then have a look at Pylons.

[Alex Martelli]

Jarek Zgoda <jz****@o2.usun .plwrote:

Daniel Nogradi napisa?(a):

For example, it HAS been published elsewhere that YouTube uses lighttpd,
not Apache: <http://trac.lighttpd.n et/trac/wiki/PoweredByLightt pd>.

How do you explain these, then:

http://www.youtube.com/results.xxx
http://www.youtube.com/results.php
http://www.youtube.com/results.py

Server signature is usually configurable.

Yeah, but I don't know why it's configured it that way. A good example
of a question that looks perfectly appropriate for YouTube's OSCON
session.
Alex

[Daniel Nogradi]

>For example, it HAS been published elsewhere that YouTube uses lighttpd,

>not Apache: <http://trac.lighttpd.n et/trac/wiki/PoweredByLightt pd>.
>
How do you explain these, then:
>
http://www.youtube.com/results.xxx
http://www.youtube.com/results.php
http://www.youtube.com/results.py

Server signature is usually configurable.

Yeah, but I don't know why it's configured it that way. A good example
of a question that looks perfectly appropriate for YouTube's OSCON
session.

Let us know what they say! :)

[John Nagle]

Bruno Desthuilliers wrote:

John Nagle a écrit :

>Victor Kryukov wrote:

>>Hello list,

our team is going to rewrite our existing web-site, which has a lot of
dynamic content and was quickly prototyped some time ago.

...

>>Our main requirement for tools we're going to use is rock-solid
stability. As one of our team-members puts it, "We want to use tools
that are stable, has many developer-years and thousands of user-years
behind them, and that we shouldn't worry about their _versions_." The
main reason for that is that we want to debug our own bugs, but not
the bugs in our tools.

You may not be happy with Python, then.

John, I'm really getting tired of your systemic and totally
unconstructive criticism. If *you* are not happy with Python, by all
means use another language.

Denying the existence of the problem won't fix it.

Many of the basic libraries for web related functions do have
problems. Even standard modules like "urllib" and "SSL" are buggy,
and have been for years. Outside the standard modules, it gets
worse, especially for ones with C components. Version incompatibility
for extensions is a serious problem. That's reality.

It's a good language, but the library situation is poor. Python as
a language is better than Perl, but CPAN is better run than Cheese Shop.

As a direct result of this, neither the Linux distro builders like
Red Hat nor major hosting providers provide Python environments that
just work. That's reality.

John Nagle

[John Nagle]

Alex Martelli wrote:

Jarek Zgoda <jz****@o2.usun .plwrote:

>>Daniel Nogradi napisa?(a):

>>>>For example, it HAS been published elsewhere that YouTube uses lighttpd,
not Apache: <http://trac.lighttpd.n et/trac/wiki/PoweredByLightt pd>.

How do you explain these, then:

http://www.youtube.com/results.xxx
http://www.youtube.com/results.php
http://www.youtube.com/results.py

Server signature is usually configurable.

Yeah, but I don't know why it's configured it that way. A good example
of a question that looks perfectly appropriate for YouTube's OSCON
session.

YouTube's home page is PHP. Try "www.youtube.co m/index.php".
That works, while the obvious alternatives don't.
If you look at the page HTML, you'll see things like

<a href="/login?next=/index.php"
onclick="_hbLin k('LogIn','Util ityLinks');">Lo g In</a>

So there's definitely PHP inside YouTube.

If you look at the HTML for YouTube pages, there seem to be two
drastically different styles. Some pages begin with "<!-- machid: 169 -->",
and have their CSS stored in external files. Those seem to be generated
by PHP. Other pages start with
"<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">",
with no "machine ID". It looks like the stuff associated with
accounts and logging in is on the second system (Python?) while the
search and view related functions are on the PHP system.

Shortly after Google bought YouTube, they replaced YouTube's search
engine (which was terrible) with one of their own. At that time,
Google search syntax, like "-", started working. That's probably
when the shift to PHP happened.

John Nagle