472,965 Members | 2,028 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,965 software developers and data experts.

urllib2 and proxies support ?

Hello all,

I've a problem using urllib2 with a proxy which need authentication.

I've tested the 'simple way' :

-- code --
import urllib

# example values for the post
my_url = 'http://www.python.org'
proxy_info = { 'host' : 'netcache.monentreprise.com',
'port' : 3128,
'user' : 'gaston.lagaffe',
'pass' : 'jeanne55'
proxy_support = urllib2.ProxyHandler({"http" :
"http://%(user)s:%(pass)s@%(host)s:%(port)d" % proxy_info})
opener = urllib2.build_opener(proxy_support)

# print proxies
print "Proxies", urllib2.getproxies()
# always print "Proxies {}" but I've set another proxy ! :-(

req = urllib2.Request(url = my_url)
handle = urllib2.urlopen(req) # get an error, seems like proxy is not
-- code --

But this doesn't work. It seems that the proxy is not recognized by
urllib2. I've read a previoust post [1] by Ray Slakinski with John Lee
answer, but unfortunatly it seems that this problem in urllib2 is well

So my questions are :
- Is there a way to make this work, and if yes, to make it work for
another user which hasn't a custom (patched) urllib2 ?
- What are your advices to use working web crawling with Python (I mean
a good support for proxies including authenticated ones, cookies, ...)
: mechanize[2] , pyCurl [3], others ?

[1] posted the 8 nov 2005 on comp.lang.python, title "urllib2 Opener
and Proxy/Authentication issues"
[2] http://wwwsearch.sourceforge.net/mechanize/
[3] http://pycurl.sourceforge.net/
Thank you
(and happy new year 2006!)

Jan 4 '06 #1
1 2716
tomazi75-nospam(at)gmail.com wrote:
Hello all,

I've a problem using urllib2 with a proxy which need authentication.

I don't have a way to test this myself but you can try the
suggestion at the bottom of this page:
Move your name/password to an HTTPBasicAuthHandler.
Jan 7 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: O. Koch | last post by:
Until now, i know that ftplib doesn't support proxies and that i have to use urllib2. But i don't know how to use the urllib2 correct. I found some examples, but i don't understand them. Is...
by: Derek Fountain | last post by:
I just tried this: >>> import urllib2 >>> urllib2.urlopen( "https://passenger.ssc.com/~dmarti/contrib-faq/" ) It sits forever. Loading that URL in a normal browser gets me a popup asking...
by: Fuzzyman | last post by:
urllib2 (under windows) will auto-detect your proxy settings and use those. Normally that's a good thing (I guess), except when it's not ! How do I switch off this behaviour ? I'm behind a...
by: Ray Slakinski | last post by:
Hello, I have defined a function to set an opener for urllib2, this opener defines any proxy and http authentication that is required. If the proxy has authencation itself and requests an...
by: Licheng Fang | last post by:
I use a HTTP proxy to connect to Internet. When I run ulropen command I get HTTP Error 407: Proxy authorization required. Could anybody tell me how to resolve this? Thanks!
by: Alejandro Dubrovsky | last post by:
I see from googling around that this is a popular topic, but I haven't seen anyone saying "ah, yes, that works", so here it goes. How does one connect through a proxy which requires basic...
by: Ju Hui | last post by:
I wanna use urllib2 to get a page with a socks 5 proxy,who can give me a sample code ? example, the proxy server is : and the port is :1080 and the username/password is :...
by: Ali.Sabil | last post by:
hello all, I just maybe hit a bug in both urllib and urllib2, actually urllib doesn't support proxy authentication, and if you setup the http_proxy env var to...
by: Astan Chee | last post by:
Hi again, According to https://demo.launchpad.net/products/python/+bug/56872 or more specifically, the example of its working code:...
by: lllomh | last post by:
Define the method first this.state = { buttonBackgroundColor: 'green', isBlinking: false, // A new status is added to identify whether the button is blinking or not } autoStart=()=>{
by: DJRhino | last post by:
Was curious if anyone else was having this same issue or not.... I was just Up/Down graded to windows 11 and now my access combo boxes are not acting right. With win 10 I could start typing...
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 4 Oct 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
by: tracyyun | last post by:
Hello everyone, I have a question and would like some advice on network connectivity. I have one computer connected to my router via WiFi, but I have two other computers that I want to be able to...
by: NeoPa | last post by:
Introduction For this article I'll be using a very simple database which has Form (clsForm) & Report (clsReport) classes that simply handle making the calling Form invisible until the Form, or all...
by: nia12 | last post by:
Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...
by: NeoPa | last post by:
Introduction For this article I'll be focusing on the Report (clsReport) class. This simply handles making the calling Form invisible until all of the Reports opened by it have been closed, when it...
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
by: GKJR | last post by:
Does anyone have a recommendation to build a standalone application to replace an Access database? I have my bookkeeping software I developed in Access that I would like to make available to other...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.