By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,903 Members | 2,036 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,903 IT Pros & Developers. It's quick & easy.

http pipelining

P: n/a
Which python module is capable of pipelining http requests?

(I know httplib can send mulitple requests per tcp connection, but in
a strictly serial way. )
Apr 26 '07 #1
Share this Question
Share on Google+
4 Replies


P: n/a
sw***@yahoo.com wrote:
Which python module is capable of pipelining http requests?

(I know httplib can send mulitple requests per tcp connection, but in
a strictly serial way. )
Oops, sorry, you meant sending requests in parallel, right?

You'll need to use either urllib or urllib2 for the web, and the
threading module is one way to run parallel requests. It's fairly easy
to use as long as you keep your tasks properly isolated form each other.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

Apr 27 '07 #2

P: n/a
sw***@yahoo.com wrote:
Which python module is capable of pipelining http requests?

(I know httplib can send mulitple requests per tcp connection, but in
a strictly serial way. )

There's nothing in the standard library, I believe, that includes both
client and server functionality in the same module. So you would need to
glue them together.

If you want a simple net proxy server I seem to remember the chameleon
system allows you to write one in about twelve lines. If it has to be
HTTP-specific, with header parsing and the like, you might want to think
about Twisted, which supports both client and server functionality and
tries to make it easy to plumb things together in pipelines.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

Apr 27 '07 #3

P: n/a
On Fri, 27 Apr 2007 13:50:21 -0400, Steve Holden <st***@holdenweb.com>
wrote:
>sw***@yahoo.com wrote:
>Which python module is capable of pipelining http requests?

(I know httplib can send mulitple requests per tcp connection, but in
a strictly serial way. )

There's nothing in the standard library, I believe, that includes both
client and server functionality in the same module. So you would need to
glue them together.

If you want a simple net proxy server I seem to remember the chameleon
system allows you to write one in about twelve lines. If it has to be
HTTP-specific, with header parsing and the like, you might want to think
about Twisted, which supports both client and server functionality and
tries to make it easy to plumb things together in pipelines.
Sorry for my confused question.

What I'm looking for is to emulate what a modern HTTP 1.1 browser like
Firefox does(when network.http.pipelining is enabled)

1.Open TCP connetion.

send multiple requests without waiting for a reply:

2.GET /url1
3.GET /url2
4.GET /url3
5.read 1st reply
6.read 2st reply
7.read 3st reply

8.Close TCP connection.

I'll be using Apache or IIS as my web server.
Apr 27 '07 #4

P: n/a
Steve Holden <st***@holdenweb.comwrites:
sw***@yahoo.com wrote:
Which python module is capable of pipelining http requests?
(I know httplib can send mulitple requests per tcp connection, but in
a strictly serial way. )
Oops, sorry, you meant sending requests in parallel, right?

You'll need to use either urllib or urllib2 for the web, and the
threading module is one way to run parallel requests. It's fairly easy
to use as long as you keep your tasks properly isolated form each
other.
No, he means "HTTP pipelining", which means sending multiple requests
down a single TCP connection, without waiting for the first response.

httplib's module-level docstring says (reformatted here):

"""
.... The HTTPResponse class does not enforce this state machine, which
implies sophisticated clients may accelerate the request/response
pipeline. Caution should be taken, though: accelerating the states
beyond the above pattern may imply knowledge of the server's
connection-close behavior for certain requests. For example, it is
impossible to tell whether the server will close the connection UNTIL
the response headers have been read; this means that further requests
cannot be placed into the pipeline until it is known that the server
will NOT be closing the connection.
"""

So, sort-of-yes, if you know what you're doing and you're lucky.

Certainly urllib and urllib2 don't support pipelining. There were
plans for a new HTTP client in Twisted with pipelining support, but I
don't know if that ever came about. AFAIK not many libraries (in any
language) support it -- e.g. none of "Jakarta commons HTTPClient",
libwww-perl, and libcurl currently support it. libwww (without the
"-perl") does claim to support it (I say "claim" merely because I
haven't used it or read the source -- no FUD intended).
Side note: As the OP mentions in a followup, by default firefox does
NOT do pipelining (to the disbelief of the people I told about this
when it came up in a previous job -- but I just tried running tcpdump
and indeed about:config seems to be telling the truth; fortunately, in
response to the limitations imposed by RFC 2616, somebody has
thoughtfully arranged for the speed of light to be fast enough for
this not to be a major problem when using websites on the other side
of the planet). The firefox people say:

http://www.mozilla.org/support/firef...oth_pipelining

"""
Pipelining is an experimental feature, designed to improve page-load
performance, that is unfortunately not well supported by some web
servers and proxies.
"""

Instead of pipelining, it uses multiple connections (2 when I tried
it, which is the maximum RFC 2616 says SHOULD be used by a
"single-user client"). I didn't try IE, but apparently it has the
same behaviour (2 connections, no pipelining):

http://blogs.msdn.com/ie/archive/2005/04/11/407189.aspx
I wonder if the right economic pressures are there for SCTP ever to
get used for everyday web HTTP stuff...
John
Apr 29 '07 #5

This discussion thread is closed

Replies have been disabled for this discussion.