By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
424,825 Members | 1,254 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 424,825 IT Pros & Developers. It's quick & easy.

Making a persistent HTTP connection

P: n/a
I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?

/David
Nov 22 '05 #1
Share this Question
Share on Google+
6 Replies


P: n/a
David Rasmussen wrote:
I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?


Are you sure HTTP supports that? This would be news to me - which
doesn't mean much :)

And even if it works - what is the problem with connections being created?

Regards,

Diez
Nov 22 '05 #2

P: n/a
Diez B. Roggisch wrote:
David Rasmussen wrote:
I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?
Are you sure HTTP supports that? This would be news to me - which
doesn't mean much :)


It does (HTTP/1.1 at least) and it's called 'keep-alive'.
And even if it works - what is the problem with connections being created?


Performance, network load...

--
Benjamin Niemann
Email: pink at odahoda dot de
WWW: http://www.odahoda.de/
Nov 22 '05 #3

P: n/a
[David Rasmussen]
I use urllib2 to do some simple HTTP communication with a web server.
In one "session", I do maybe 10-15 requests. It seems that urllib2
opens op a connection every time I do a request. Can I somehow make it
use _one_ persistent connection where I can do multiple GET->"receive
data" passes before the connection is closed?

[Diez B. Roggisch] Are you sure HTTP supports that?
Yes, HTTP 1.1 definitely supports multiple requests on the same connection.

http://www.w3.org/Protocols/rfc2616/...c8.html#sec8.1

Some HTTP 1.0 clients supported persistent connections through the use
of the non-standard "keep-alive" header.
And even if it works - what is the problem with connections being created?


The URL above describes the benefits of persistent connections. The
primary problem of the old style of one-request-per-connection is the
creation of more sockets than are necessary.

To the OP: neither urllib nor urllib2 implements persistent connections,
but httplib does. See the httplib documentation page for an example.

http://www.python.org/doc/2.4.2/lib/...-examples.html

However, even httplib is "synchronous", in that it cannot pipeline
requests: the response to the first request must be competely read
before a second request can be issued.

HTH,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan
Nov 22 '05 #4

P: n/a
[David Rasmussen]
I use urllib2 to do some simple HTTP communication with a web server.
In one "session", I do maybe 10-15 requests. It seems that urllib2
opens op a connection every time I do a request. Can I somehow make it
use _one_ persistent connection where I can do multiple GET->"receive
data" passes before the connection is closed?

[Diez B. Roggisch] Are you sure HTTP supports that?
Yes, HTTP 1.1 definitely supports multiple requests on the same
connection.

http://www.w3.org/Protocols/rfc2616/...c8.html#sec8.1

Some HTTP 1.0 clients supported persistent connections through the use
of the non-standard "keep-alive" header.
And even if it works - what is the problem with connections being created?


The URL above describes the benefits of persistent connections. The
primary problem of the old style of one-request-per-connection is the
creation of more sockets than are necessary.

To the OP: neither urllib nor urllib2 implements persistent
connections, but httplib does. See the httplib documentation page for
an example.

http://www.python.org/doc/2.4.2/lib/...-examples.html

However, even httplib is "synchronous", in that it cannot pipeline
requests: the response to the first request must be competely read
before a second request can be issued.

HTH,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

Nov 22 '05 #5

P: n/a
>>>>> "Alan Kennedy" <al****@hotmail.com> (AK) wrote:
AK> http://www.w3.org/Protocols/rfc2616/...c8.html#sec8.1 AK> Some HTTP 1.0 clients supported persistent connections through the use
AK> of the non-standard "keep-alive" header.
And even if it works - what is the problem with connections being created?

AK> The URL above describes the benefits of persistent connections. The
AK> primary problem of the old style of one-request-per-connection is the
AK> creation of more sockets than are necessary.


Maybe even more important (and just briefly mentioned in the section
referred to above) is the latency introduced by the TCP setup and the slow
startup phase of TCP's congestion control. This calculation is one of the
exercises the students have to make in my networks class.
--
Piet van Oostrum <pi**@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: pi**@vanoostrum.org
Nov 22 '05 #6

P: n/a
>>>>> "Alan Kennedy" <al****@hotmail.com> (AK) wrote:
AK> http://www.w3.org/Protocols/rfc2616/...c8.html#sec8.1 AK> Some HTTP 1.0 clients supported persistent connections through the use
AK> of the non-standard "keep-alive" header.
And even if it works - what is the problem with connections being created?

AK> The URL above describes the benefits of persistent connections. The
AK> primary problem of the old style of one-request-per-connection is the
AK> creation of more sockets than are necessary.


Maybe even more important (and just briefly mentioned in the section
referred to above) is the latency introduced by the TCP setup and the slow
startup phase of TCP's congestion control. This calculation is one of the
exercises the students have to make in my networks class.
--
Piet van Oostrum <pi**@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: pi**@vanoostrum.org
Nov 22 '05 #7

This discussion thread is closed

Replies have been disabled for this discussion.