468,133 Members | 1,623 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,133 developers. It's quick & easy.

urllib2 and proxies support ?

Hello all,

I've a problem using urllib2 with a proxy which need authentication.

I've tested the 'simple way' :

-- code --
import urllib

# example values for the post
my_url = 'http://www.python.org'
proxy_info = { 'host' : 'netcache.monentreprise.com',
'port' : 3128,
'user' : 'gaston.lagaffe',
'pass' : 'jeanne55'
}
proxy_support = urllib2.ProxyHandler({"http" :
"http://%(user)s:%(pass)s@%(host)s:%(port)d" % proxy_info})
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)

# print proxies
print "Proxies", urllib2.getproxies()
# always print "Proxies {}" but I've set another proxy ! :-(

req = urllib2.Request(url = my_url)
handle = urllib2.urlopen(req) # get an error, seems like proxy is not
contacted
-- code --

But this doesn't work. It seems that the proxy is not recognized by
urllib2. I've read a previoust post [1] by Ray Slakinski with John Lee
answer, but unfortunatly it seems that this problem in urllib2 is well
know.

So my questions are :
- Is there a way to make this work, and if yes, to make it work for
another user which hasn't a custom (patched) urllib2 ?
- What are your advices to use working web crawling with Python (I mean
a good support for proxies including authenticated ones, cookies, ...)
: mechanize[2] , pyCurl [3], others ?

[1] posted the 8 nov 2005 on comp.lang.python, title "urllib2 Opener
and Proxy/Authentication issues"
[2] http://wwwsearch.sourceforge.net/mechanize/
[3] http://pycurl.sourceforge.net/
Thank you
(and happy new year 2006!)
Tom

Jan 4 '06 #1
1 2509
DH
tomazi75-nospam(at)gmail.com wrote:
Hello all,

I've a problem using urllib2 with a proxy which need authentication.


I don't have a way to test this myself but you can try the
suggestion at the bottom of this page:
http://aspn.activestate.com/ASPN/Coo...n/Recipe/52199
Move your name/password to an HTTPBasicAuthHandler.
Jan 7 '06 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

4 posts views Thread by O. Koch | last post: by
2 posts views Thread by Derek Fountain | last post: by
4 posts views Thread by Fuzzyman | last post: by
1 post views Thread by Ray Slakinski | last post: by
6 posts views Thread by Alejandro Dubrovsky | last post: by
reply views Thread by Ali.Sabil | last post: by
1 post views Thread by Astan Chee | last post: by
27 posts views Thread by didacticone | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.