468,140 Members | 1,165 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,140 developers. It's quick & easy.

Python equivalent of LWP and HTTP in Perl

I have a script that I've written in Perl that retrieves files generated
from a template. It works decently enough, but I'd like to rewrite it in
Python (Perl was just a detour; it was originally Sed).

Was wondering what the closest thing to what I'm using now in Perl
(LWP::UserAgent and HTTP::Request:Common) is in Python. The main bit of
my code is fairly simple.

my($ua) = LWP::UserAgent->new;
my $req = GET "$uri";
$req->header(Referer => "$referer");
$ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
$response = $ua->request($req);
$respcode = $response->code;

Jul 18 '05 #1
4 7496
Dekaritae wrote:
I have a script that I've written in Perl that retrieves files generated
from a template. It works decently enough, but I'd like to rewrite it in
Python (Perl was just a detour; it was originally Sed). my($ua) = LWP::UserAgent->new;
my $req = GET "$uri";
$req->header(Referer => "$referer");
$ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
$response = $ua->request($req);
$respcode = $response->code;

import urllib
proxies = {'http': 'http://localhost:8080/'}
response = urllib.urlopen("some_url", proxies=proxies).read()

--Irmen
Jul 18 '05 #2

"Irmen de Jong" wrote
my($ua) = LWP::UserAgent->new;
my $req = GET "$uri";
$req->header(Referer => "$referer");
$ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
$response = $ua->request($req);
$respcode = $response->code;

import urllib
proxies = {'http': 'http://localhost:8080/'}
response = urllib.urlopen("some_url", proxies=proxies).read()


.... which is short and clean, but ignores the Referer header.
I have worked with httplib some, but not urllib. Maybe urllib can do this,
but may I suggest checking out httplib:
http://docs.python.org/lib/httplib-examples.html

-ej
Jul 18 '05 #3
"Erik Johnson" <sp**@nospam.org> writes:
"Irmen de Jong" wrote
> my($ua) = LWP::UserAgent->new;
> my $req = GET "$uri";
> $req->header(Referer => "$referer");
> $ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
> $response = $ua->request($req);
> $respcode = $response->code;

import urllib
proxies = {'http': 'http://localhost:8080/'}
response = urllib.urlopen("some_url", proxies=proxies).read()


... which is short and clean, but ignores the Referer header.
I have worked with httplib some, but not urllib. Maybe urllib can do this,
but may I suggest checking out httplib:
http://docs.python.org/lib/httplib-examples.html


Try urllib2:

import urllib2

proxy_support = urllib2.ProxyHandler({"http" : "http://localhost:8080/"})
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)

req = urllib2.Request("some_url")
req.add_header("Referer", referer)
response = urllib2.urlopen(req)
There's fancier handlers too, if you need digest authentication, for
example. If you have the environment variable 'http_proxy' set to your
proxy, you don't need to do the install_opener jazz.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca
Jul 18 '05 #4
Dekaritae <ma**@mdi.ca> wrote in message news:<vA*************@fe61.usenetserver.com>...
I have a script that I've written in Perl that retrieves files generated
from a template. It works decently enough, but I'd like to rewrite it in
Python (Perl was just a detour; it was originally Sed).

Was wondering what the closest thing to what I'm using now in Perl
(LWP::UserAgent and HTTP::Request:Common) is in Python. The main bit of
my code is fairly simple.

my($ua) = LWP::UserAgent->new;
my $req = GET "$uri";
$req->header(Referer => "$referer");
$ua->proxy('http', 'http://localhost:8080/'); # Proxomitron
$response = $ua->request($req);
$respcode = $response->code;


urllib2 is definitely the way forward for adding proxy support and
request headers. Very easy. If you want cookie handling you'll need to
look at ClientCookie which does similar things to the perl library you
mention.

Regards,

Fuzzy
http://www.voidspace.org.uk/atlantib...thonutils.html
Jul 18 '05 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

54 posts views Thread by Brandon J. Van Every | last post: by
699 posts views Thread by mike420 | last post: by
226 posts views Thread by Stephen C. Waterbury | last post: by
1 post views Thread by Edward WIJAYA | last post: by
68 posts views Thread by Lad | last post: by
20 posts views Thread by Xah Lee | last post: by
reply views Thread by Xah Lee | last post: by
14 posts views Thread by ccdetail | last post: by
10 posts views Thread by Mladen Gogala | last post: by
27 posts views Thread by didacticone | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.