High volume websites using Python web server software?

[Irmen de Jong]

Hi,
Things like Twisted, medusa, etc.... that claim to be able
to support hundreds of concurrent connections because of the
async I/O framework they're based on....
can someone give a few examples of some web sites actually
using those Python frameworks?

Thanks :)

--Irmen

[Valentino Volonghi aka Dialtone]

Irmen de Jong <[e-mail]> wrote:

Things like Twisted, medusa, etc.... that claim to be able
to support hundreds of concurrent connections because of the
async I/O framework they're based on....
can someone give a few examples of some web sites actually
using those Python frameworks?

http://www.diadi.it
http://www.superleague.co.uk
http://www.playzone.cz
http://www.divmod.com
http://www.divmod.org

These are those that I know made with Twisted + Nevow

There are others obviously. And other people will tell you about sites made
with other python frameworks.

BTW: I'm writing a forum with Nevow, and from the last tests I've run, I can
tell you that with a normal page (3 db queries) and no caching or
optimizations, I can handle 600.000 connections/day which is almost
20M/month.
I've used ab to measure this performances and I think they are good ones.

--
Valentino Volonghi aka Dialtone
Now running FreeBSD 5.3-beta7
Blog: http://vvolonghi.blogspot.com

[Wilk]

Irmen de Jong <[e-mail]> writes:

Hi,
Things like Twisted, medusa, etc.... that claim to be able
to support hundreds of concurrent connections because of the
async I/O framework they're based on....
can someone give a few examples of some web sites actually
using those Python frameworks?

online scrabble game : http://seps.flibuste.net with more than 650.000
hits/day

I used twisted to can use the jabber and telnet protocol.

--
Wilk - http://flibuste.net

[Larry Bates]

Irmen de Jong wrote:

Hi,
Things like Twisted, medusa, etc.... that claim to be able
to support hundreds of concurrent connections because of the
async I/O framework they're based on....
can someone give a few examples of some web sites actually
using those Python frameworks?

Thanks :)

--Irmen

I don't know if it is of interested but it is my understanding
that Zope uses some version of Medusa and there are quite a few
"high volume" websites based on zope (see www.zope.com).

Larry Bates

[Michael Sparks]

Irmen de Jong wrote:

Hi,
Things like Twisted, medusa, etc.... that claim to be able
to support hundreds of concurrent connections because of the
async I/O framework they're based on....
can someone give a few examples of some web sites actually
using those Python frameworks?

What do you define as high volume? Different people have different
expectations here. I would say for example that high volume starts at
around 10-20 million requests per day, but even that is an order of
magnitude too low to *really* be viewed as high volume, _unless_ you're
only counting valid page impressions and not all web objects served.

That said at that level you're generally dealing with a server farm,
and no matter how big you think *your* site is, there's (almost) always
someone bigger *especially* IF you factor in web proxy systems as being
web server software as well.

Also do you mean python application systems on top of web servers or
python as the web server - sounds like the latter? (As far as I'm aware
the BBC don't use python in either area, but I could be wrong, but these
questions sprang to mind reading the responses you've had from others).

Regards,
Michael.
--
Mi************@rd.bbc.co.uk
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

[Ian Bicking]

Michael Sparks wrote:

Irmen de Jong wrote:

Hi,
Things like Twisted, medusa, etc.... that claim to be able
to support hundreds of concurrent connections because of the
async I/O framework they're based on....
can someone give a few examples of some web sites actually
using those Python frameworks?

What do you define as high volume? Different people have different
expectations here. I would say for example that high volume starts at
around 10-20 million requests per day, but even that is an order of
magnitude too low to *really* be viewed as high volume, _unless_ you're
only counting valid page impressions and not all web objects served.

I think he's talking about servers that server a higher-than-normal
number of simultaneous connections.

For instance, with Apache you can only serve as many connections as you
have processes or threads. By default, that usually leaves you with
maybe ten concurrent connections (with everyone else being queued), and
while you can bump that up considerably there's still a limit.

This works fine when requests are quickly completed, because you can
still serve a large number of requests even if only a small number of
concurrent request. But if the requests are long-running (e.g., a
telnet session) then you need to serve a large number of requests.
Because async frameworks don't have the overhead of multiple processes
or threads, they can potentially serve a large number of simultaneous
requests where other systems cannot.

--
Ian Bicking / ia**@colorstudy.com / http://blog.ianbicking.org

[Irmen de Jong]

Ian Bicking wrote:

I think he's talking about servers that server a higher-than-normal
number of simultaneous connections.

Well, not necessarily so, but this is an even better description.
I was looking for web sites with moderate visitor counts
(where 'high' means Yahoo, Google, Cnn, and 'low' means a
200-user forum about automobiles) that are running on one of Python's
web application servers / network servers.

If it's a site that uses an async I/O framework, and it actually
does sustains the potential very large amount of simultaneous
connections, even better.
Actually, it doesn't even have to be a web site. A 10.000-user IRC
server in Python would also qualify :)

Why am I asking this?
Someone on a programming forum elsewhere was sceptical that you
can create high performance Python network servers.
I suggested Twisted or Medusa. He asked for example servers.
So there you are :)
Anyway I've got a few pointers from the replies already given.
Thanks!
--Irmen.

[Josiah Carlson]

Irmen de Jong <[e-mail]> wrote:

Ian Bicking wrote:

I think he's talking about servers that server a higher-than-normal
number of simultaneous connections.

Well, not necessarily so, but this is an even better description.
I was looking for web sites with moderate visitor counts
(where 'high' means Yahoo, Google, Cnn, and 'low' means a
200-user forum about automobiles) that are running on one of Python's
web application servers / network servers.

If it's a site that uses an async I/O framework, and it actually
does sustains the potential very large amount of simultaneous
connections, even better.
Actually, it doesn't even have to be a web site. A 10.000-user IRC
server in Python would also qualify :)

Why am I asking this?
Someone on a programming forum elsewhere was sceptical that you
can create high performance Python network servers.
I suggested Twisted or Medusa. He asked for example servers.
So there you are :)


Anyway I've got a few pointers from the replies already given.
Thanks!

Using a heavily customized variant of asyncore, I have written a client
that can handle 500 simultaneous connections (default file handle limit
with Python is 512) on a modern (1+ghz) processor. It is actually
limited by bandwidth and latency on our 100mbit connection, as the
protocol is fairly chatty.

We have benchmarked it (each command results in a delay of
.25/random.random() seconds, timeouts set at 30 seconds), and it comes
out to roughly 4.8 million of these connections per day on a single
processor (we are unsure if it scales running multiple processes on a
multiple processor machine, as 99% of the processor use is system-level
select.select() calls).

We are actually concerned with accidentally DOSing remote servers that
have less efficient server implementations.

Is that high enough volume?
- Josiah