Making a persistent HTTP connection

[David Rasmussen]

I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?

/David

[Diez B. Roggisch]

David Rasmussen wrote:

I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?

Are you sure HTTP supports that? This would be news to me - which
doesn't mean much :)

And even if it works - what is the problem with connections being created?

Regards,

Diez

[Benjamin Niemann]

Diez B. Roggisch wrote:

David Rasmussen wrote:

I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?
Are you sure HTTP supports that? This would be news to me - which
doesn't mean much :)

It does (HTTP/1.1 at least) and it's called 'keep-alive'.
And even if it works - what is the problem with connections being created?

Performance, network load...

--
Benjamin Niemann
Email: pink at odahoda dot de
WWW: http://www.odahoda.de/

[Alan Kennedy]

[David Rasmussen]

I use urllib2 to do some simple HTTP communication with a web server.
In one "session", I do maybe 10-15 requests. It seems that urllib2
opens op a connection every time I do a request. Can I somehow make it
use _one_ persistent connection where I can do multiple GET->"receive
data" passes before the connection is closed?

[Diez B. Roggisch] Are you sure HTTP supports that?
Yes, HTTP 1.1 definitely supports multiple requests on the same connection.

http://www.w3.org/Protocols/rfc2616/...c8.html#sec8.1

Some HTTP 1.0 clients supported persistent connections through the use
of the non-standard "keep-alive" header.
And even if it works - what is the problem with connections being created?

The URL above describes the benefits of persistent connections. The
primary problem of the old style of one-request-per-connection is the
creation of more sockets than are necessary.

To the OP: neither urllib nor urllib2 implements persistent connections,
but httplib does. See the httplib documentation page for an example.

http://www.python.org/doc/2.4.2/lib/...-examples.html

However, even httplib is "synchronous", in that it cannot pipeline
requests: the response to the first request must be competely read
before a second request can be issued.

HTH,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

[Alan Kennedy]

[David Rasmussen]

I use urllib2 to do some simple HTTP communication with a web server.
In one "session", I do maybe 10-15 requests. It seems that urllib2
opens op a connection every time I do a request. Can I somehow make it
use _one_ persistent connection where I can do multiple GET->"receive
data" passes before the connection is closed?

[Diez B. Roggisch] Are you sure HTTP supports that?
Yes, HTTP 1.1 definitely supports multiple requests on the same
connection.

http://www.w3.org/Protocols/rfc2616/...c8.html#sec8.1

Some HTTP 1.0 clients supported persistent connections through the use
of the non-standard "keep-alive" header.
And even if it works - what is the problem with connections being created?

The URL above describes the benefits of persistent connections. The
primary problem of the old style of one-request-per-connection is the
creation of more sockets than are necessary.

To the OP: neither urllib nor urllib2 implements persistent
connections, but httplib does. See the httplib documentation page for
an example.

http://www.python.org/doc/2.4.2/lib/...-examples.html

However, even httplib is "synchronous", in that it cannot pipeline
requests: the response to the first request must be competely read
before a second request can be issued.

HTH,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

[Piet van Oostrum]

>>>>> "Alan Kennedy" <al****@hotmail.com> (AK) wrote:

AK> http://www.w3.org/Protocols/rfc2616/...c8.html#sec8.1 AK> Some HTTP 1.0 clients supported persistent connections through the use
AK> of the non-standard "keep-alive" header.

And even if it works - what is the problem with connections being created?

AK> The URL above describes the benefits of persistent connections. The
AK> primary problem of the old style of one-request-per-connection is the
AK> creation of more sockets than are necessary.

Maybe even more important (and just briefly mentioned in the section
referred to above) is the latency introduced by the TCP setup and the slow
startup phase of TCP's congestion control. This calculation is one of the
exercises the students have to make in my networks class.
--
Piet van Oostrum <pi**@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: pi**@vanoostrum.org

[Piet van Oostrum]

>>>>> "Alan Kennedy" <al****@hotmail.com> (AK) wrote:

AK> http://www.w3.org/Protocols/rfc2616/...c8.html#sec8.1 AK> Some HTTP 1.0 clients supported persistent connections through the use
AK> of the non-standard "keep-alive" header.

And even if it works - what is the problem with connections being created?

AK> The URL above describes the benefits of persistent connections. The
AK> primary problem of the old style of one-request-per-connection is the
AK> creation of more sockets than are necessary.

Maybe even more important (and just briefly mentioned in the section
referred to above) is the latency introduced by the TCP setup and the slow
startup phase of TCP's congestion control. This calculation is one of the
exercises the students have to make in my networks class.
--
Piet van Oostrum <pi**@cs.uu.nl>
URL: http://www.cs.uu.nl/~piet [PGP 8DAE142BE17999C4]
Private email: pi**@vanoostrum.org