468,107 Members | 1,303 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,107 developers. It's quick & easy.

Using a proxy with urllib2

I'm trying to use a proxy server with urllib2.
So I have managed to get it to work by setting the environment
variable:
export HTTP_PROXY=127.0.0.1:8081

But I wanted to set it from the code. However, this does not set the proxy:
httpproxy = '127.0.0.1:3129'
proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
urllib2.install_opener(opener)
I'm using it from a web.py URL handler file, not sure if it matters.

I have another question though. It seems that using either of the
methods above, the proxy will be global. What if I want to use
a proxy with one site, but not with another site? Or even use a
proxy for some URLs but not others? The proxy having to be global
is really not convenient. Is there any way to do per-fetch proxy?
Jan 10 '08 #1
6 2716
"Jack" <no****@invalid.comwrites:
I'm trying to use a proxy server with urllib2.
So I have managed to get it to work by setting the environment
variable:
export HTTP_PROXY=127.0.0.1:8081

But I wanted to set it from the code. However, this does not set the proxy:
httpproxy = '127.0.0.1:3129'
proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
urllib2.install_opener(opener)
Works for me.
How do you know that the proxy is not set?
I'm using it from a web.py URL handler file, not sure if it matters.
I don't think so.
I have another question though. It seems that using either of the
methods above, the proxy will be global. What if I want to use
a proxy with one site, but not with another site? Or even use a
proxy for some URLs but not others? The proxy having to be global
is really not convenient. Is there any way to do per-fetch proxy?
Try this:

<code>
import urllib2

def getopener(proxy=None):
opener = urllib2.build_opener(urllib2.HTTPHandler)
if proxy:
proxy_support = urllib2.ProxyHandler({"http": "http://" + proxy})
opener.add_handler(proxy_support)
return opener

def fetchurl(url, opener):
f = opener.open(url)
data = f.read()
f.close()
return data

print fetchurl('http://www.python.org', getopener('127.0.0.1:8081'))
</code>

HTH,
Rob
Jan 10 '08 #2
Works for me.
How do you know that the proxy is not set?
The proxy drops some URLs and the URLs were not being dropped when I did
this :)
Try this:
Thank you. I'll give it a try.
Jan 11 '08 #3
Rob,

I tried your code snippet and it worked great. I'm just wondering if
getopener( ) call
is lightweight so I can just call it in every call to fetchurl( )? Or I
should try to share
the opener object among fetchurl( ) calls?

Thanks,
Jack
"Rob Wolfe" <rw@smsnet.plwrote in message
news:87************@merkury.smsnet.pl...
Try this:

<code>
import urllib2

def getopener(proxy=None):
opener = urllib2.build_opener(urllib2.HTTPHandler)
if proxy:
proxy_support = urllib2.ProxyHandler({"http": "http://" + proxy})
opener.add_handler(proxy_support)
return opener

def fetchurl(url, opener):
f = opener.open(url)
data = f.read()
f.close()
return data

print fetchurl('http://www.python.org', getopener('127.0.0.1:8081'))
</code>

HTH,
Rob

Jan 11 '08 #4
"Jack" <no****@invalid.comwrites:
Rob,

I tried your code snippet and it worked great. I'm just wondering if
getopener( ) call
is lightweight so I can just call it in every call to fetchurl( )? Or I
should try to share
the opener object among fetchurl( ) calls?
Creating an opener for every url is rather not reasonable way to go.
Sharing is the better approach. In your case you might
create e.g. two instances: simple_opener and proxy_opener.

Regards,
Rob
Jan 11 '08 #5
>I'm trying to use a proxy server with urllib2.
So I have managed to get it to work by setting the environment
variable:
export HTTP_PROXY=127.0.0.1:8081

But I wanted to set it from the code. However, this does not set the
proxy:
httpproxy = '127.0.0.1:3129'
proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
urllib2.install_opener(opener)
I find out why it doesn't work in my code but I don't have a solution -
somewhere
else in the code calls these two lines:

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(c j))
urllib2.install_opener(opener)

and they override the proxy opener. Could anyone tell me how to use both
openers?
Jan 12 '08 #6
"Jack" <no****@invalid.comwrites:
>>I'm trying to use a proxy server with urllib2.
So I have managed to get it to work by setting the environment
variable:
export HTTP_PROXY=127.0.0.1:8081

But I wanted to set it from the code. However, this does not set the
proxy:
httpproxy = '127.0.0.1:3129'
proxy_support = urllib2.ProxyHandler({"http":"http://" + httpproxy})
opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler)
urllib2.install_opener(opener)

I find out why it doesn't work in my code but I don't have a solution -
somewhere
else in the code calls these two lines:

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(c j))
urllib2.install_opener(opener)

and they override the proxy opener. Could anyone tell me how to use both
openers?
You don't have to create another opener if you only want to add
some handler. You can use `add_handler` method, e.g.:
opener.add_handler(urllib2.HTTPCookieProcessor(cj) )

HTH,
Rob

Jan 12 '08 #7

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

4 posts views Thread by O. Koch | last post: by
2 posts views Thread by Andre Bocchini | last post: by
4 posts views Thread by bmiras | last post: by
reply views Thread by Lee, Dustin | last post: by
4 posts views Thread by Fuzzyman | last post: by
1 post views Thread by Ray Slakinski | last post: by
3 posts views Thread by itay_k | last post: by
1 post views Thread by Magnus.Moraberg | last post: by
1 post views Thread by Solo | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.