urllib2 and proxies support ?

[tomazi75-nospam(at)gmail.com]

Hello all,

I've a problem using urllib2 with a proxy which need authentication.

I've tested the 'simple way' :

-- code --
import urllib

# example values for the post
my_url = 'http://www.python.org'
proxy_info = { 'host' : 'netcache.monentreprise.com',
'port' : 3128,
'user' : 'gaston.lagaffe',
'pass' : 'jeanne55'
}
proxy_support = urllib2.ProxyHandler({"http" :
"http://%(user)s:%(pass)s@%(host)s:%(port)d" % proxy_info})
opener = urllib2.build_opener(proxy_support)
urllib2.install_opener(opener)

# print proxies
print "Proxies", urllib2.getproxies()
# always print "Proxies {}" but I've set another proxy ! :-(

req = urllib2.Request(url = my_url)
handle = urllib2.urlopen(req) # get an error, seems like proxy is not
contacted
-- code --

But this doesn't work. It seems that the proxy is not recognized by
urllib2. I've read a previoust post [1] by Ray Slakinski with John Lee
answer, but unfortunatly it seems that this problem in urllib2 is well
know.

So my questions are :
- Is there a way to make this work, and if yes, to make it work for
another user which hasn't a custom (patched) urllib2 ?
- What are your advices to use working web crawling with Python (I mean
a good support for proxies including authenticated ones, cookies, ...)
: mechanize[2] , pyCurl [3], others ?

[1] posted the 8 nov 2005 on comp.lang.python, title "urllib2 Opener
and Proxy/Authentication issues"
[2] http://wwwsearch.sourceforge.net/mechanize/
[3] http://pycurl.sourceforge.net/
Thank you
(and happy new year 2006!)
Tom

[DH]

tomazi75-nospam(at)gmail.com wrote:

Hello all,

I've a problem using urllib2 with a proxy which need authentication.

I don't have a way to test this myself but you can try the
suggestion at the bottom of this page:
http://aspn.activestate.com/ASPN/Coo...n/Recipe/52199
Move your name/password to an HTTPBasicAuthHandler.