Python as a HTTP Client

[pinkfloydhomer]

I am writing a program that has to do some lightweight HTTP
communication with a webserver on the internet. I haven't checked, but
I'm sure I could do something lowlevel like opening a socket myself and
then send/receive everything myself on this (how do I do that?), but
I'd bet that Python have some module which is more high level.
Something that would just let me connect using an URL, send a few GETs,
and receive the answer as a string/file etc.

Does this exist, and where can I read about it?

/David

[James Tanis]

If you haven't discovered www.python.org yet I suggest going there :P.
You will find there the documentation you need under the conspicuous
name library reference. Specifically the modules you'd probably most
be interested in are urllib/urllib2/httplib depending on what you
need. Their may be other external modules which fit your task even
better, try doing a search through the Python Package Index..
On 10 Nov 2005 05:02:32 -0800, pi************@gmail.com
<pi************@gmail.com> wrote:

I am writing a program that has to do some lightweight HTTP
communication with a webserver on the internet. I haven't checked, but
I'm sure I could do something lowlevel like opening a socket myself and
then send/receive everything myself on this (how do I do that?), but
I'd bet that Python have some module which is more high level.
Something that would just let me connect using an URL, send a few GETs,
and receive the answer as a string/file etc.

Does this exist, and where can I read about it?

/David

--
http://mail.python.org/mailman/listinfo/python-list

--
James Tanis
jt****@pycoder.org
http://pycoder.org

[David Rasmussen]

James Tanis wrote:

If you haven't discovered www.python.org yet I suggest going there :P.
You will find there the documentation you need under the conspicuous
name library reference. Specifically the modules you'd probably most
be interested in are urllib/urllib2/httplib depending on what you
need. Their may be other external modules which fit your task even
better, try doing a search through the Python Package Index..

To both you and Frederik:

I do know about www.python.org. I do an extensive amount of googling in
general and searching at python.org before I ask questions such as this.
I did stumble upon urllib, urllib2 and httplib in the documentation, but
let me assure you, as a newbie, that finding this documentation doesn't
make one go "ah, this is what I was looking for". Specifically, I can't
see from reference documentation whether something even smarter or more
highlevel exists. Fuzzyman's link did the trick. It also helped me
(after reading his articles) to understand the reference documentation
better.

/David

[Fredrik Lundh]

David Rasmussen wrote:

I do know about www.python.org. I do an extensive amount of googling in
general and searching at python.org before I ask questions such as this.
I did stumble upon urllib, urllib2 and httplib in the documentation, but
let me assure you, as a newbie, that finding this documentation doesn't
make one go "ah, this is what I was looking for". Specifically, I can't see from reference documentation whether something
even smarter or more highlevel exists.

hmm. so if that was your question, why did you write:

I am writing a program that has to do some lightweight HTTP
communication with a webserver on the internet. I haven't checked, but
I'm sure I could do something lowlevel like opening a socket myself and
then send/receive everything myself on this (how do I do that?), but
I'd bet that Python have some module which is more high level.
Something that would just let me connect using an URL, send a few GETs,
and receive the answer as a string/file etc.

? ("I haven't checked ... but I'd bet" doesn't really sound like "I've checked
the docs and found a couple of modules that seem to do this, but I wonder
if there is something better out there")

....especially if you had already seen the tutorial's

Internet Protocols

There are a number of modules for accessing the internet and processing
internet protocols. Two of the simplest are urllib2 for retrieving data
from urls /.../

(followed by a brief example that shows how to read from an URL)

or the reference guide's

urllib -- Open arbitrary resources by URL

This module provides a high-level interface for fetching data across the
World Wide Web. In particular, the urlopen() function is similar to the built-
in function open(), but accepts Universal Resource Locators (URLs) instead
of filenames.

or

urllib2 -- extensible library for opening URLs

The urllib2 module defines functions and classes which help in opening URLs
(mostly HTTP) in a complex world -- basic and digest authentication, re-
directions, cookies and more.

which all seem to match "something that would just let me connect
using an URL" pretty well.

</F>