472,372 Members | 1,955 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,372 software developers and data experts.

FTP with urllib2 behind a proxy

Until now, i know that ftplib doesn't support proxies and that i have
to use urllib2. But i don't know how to use the urllib2 correct. I
found some examples, but i don't understand them.

Is there anyone who can help me?
Jul 18 '05 #1
4 19071
You need to install a proxyhandler, authhandler,
ftphandler and httphandler. Then build yourself
an opener, that opens the doors for you ... :-)

The following does the trick.

proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
'https' : 'myhttpsproxy:443',
'ftp' : 'myftpproxy:21' } )

opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
urllib2.HTTPHandler, urllib2.HTTPSHandler,
urllib2.FTPHandler)

# install this opener
urllib2.install_opener(opener)

# Go ahead, knock knock!

req=urlli2.Request('ftp://ftp.gnu.org')
data=urllib2.urlopen(req).read()

Of course, replace the arbit proxy values I wrote
with your proxy values. If your proxy need authentication
, you will need to do a bit more here.

proxyauth='http://' + username + '@' + password + 'myproxy:myproxport'

Then your proxy handler becomes ( I am assuming a generic proxy
for all protocols here!)

proxy_handler = urllib2.ProxyHandler ( {'http' : proxyauth,
'https' : proxyauth,
'ftp' : proxyauth } )

HTH.

-Anand
jj*@pobox.com (John J. Lee) wrote in message news:<87************@pobox.com>...
Za***********@gmx.de (O. Koch) writes:
Until now, i know that ftplib doesn't support proxies and that i have
to use urllib2. But i don't know how to use the urllib2 correct. I
found some examples, but i don't understand them.

Is there anyone who can help me?


import urllib2
response = urllib2.urlopen("ftp://ftp.example.com/pub/myfile")
data = response.read()
response.close()
Does that do the trick?
John

Jul 18 '05 #2
py*******@Hotpop.com (Anand Pillai) writes:
You need to install a proxyhandler, authhandler,
ftphandler and httphandler. Then build yourself
an opener, that opens the doors for you ... :-)

The following does the trick.

proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
'https' : 'myhttpsproxy:443',
'ftp' : 'myftpproxy:21' } )

opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
urllib2.HTTPHandler, urllib2.HTTPSHandler,
urllib2.FTPHandler)

# install this opener
urllib2.install_opener(opener)

# Go ahead, knock knock!

req=urlli2.Request('ftp://ftp.gnu.org')
data=urllib2.urlopen(req).read()


A couple of things to add: you don't need to add handlers that already
get added by default by build_opener (FTPHandler and HTTPHandler, for
example). ProxyHandler is one of these default handlers, so if your
environment is set up for it (http_proxy, etc. environment variables),
you don't need to supply a ProxyHandler (of course, if your
environ. *isn't*, then you do need to supply one, to give it the proxy
details). You don't need Request objects (unless you want to add
headers to a Request, or pass Requests around). You don't need to
install a global opener, unless your code expects it -- it's just a
convenience (or an inconvenience, sometimes).

Actually, did the OP say proxy basic auth. was involved? Don't
recall. I've never needed it for proxies, but there seems to be a
ProxyBasicAuthHandler in urllib2, so I guess that's what you meant to
use, rather than HTTPBasicAuthHandler (which is for website auth., not
proxy auth).

So, after all that, you end up with:

opener = urllib2.build_opener(urllib2.ProxyBasicAuthHandler )
data = opener.open('ftp://ftp.gnu.org').read()

(I like to close the response explicitly, though)
John
Jul 18 '05 #3
jj*@pobox.com (John J. Lee) wrote in message news:<87************@pobox.com>...
py*******@Hotpop.com (Anand Pillai) writes:

The code obviously caters to the bottom-line, in case
the programmer does not want to worry about environment
variables, would like to install the handlers himself
and his proxy needs authentication.

In the last case, the HTTP_PROXY env variable does not
help and you need to install the proxy handler yourself.

I copied this stuff from my program which needs all these
and more ( a USER-AGENT header for example ).

-Anand
You need to install a proxyhandler, authhandler,
ftphandler and httphandler. Then build yourself
an opener, that opens the doors for you ... :-)

The following does the trick.

proxy_handler = urllib2.ProxyHandler( {'http': 'myhttpproxy:80',
'https' : 'myhttpsproxy:443',
'ftp' : 'myftpproxy:21' } )

opener= urllib2.build_opener(proxy_handler, urllib2.HTTPBasicAuthHandler(),
urllib2.HTTPHandler, urllib2.HTTPSHandler,
urllib2.FTPHandler)

# install this opener
urllib2.install_opener(opener)

# Go ahead, knock knock!

req=urlli2.Request('ftp://ftp.gnu.org')
data=urllib2.urlopen(req).read()


A couple of things to add: you don't need to add handlers that already
get added by default by build_opener (FTPHandler and HTTPHandler, for
example). ProxyHandler is one of these default handlers, so if your
environment is set up for it (http_proxy, etc. environment variables),
you don't need to supply a ProxyHandler (of course, if your
environ. *isn't*, then you do need to supply one, to give it the proxy
details). You don't need Request objects (unless you want to add
headers to a Request, or pass Requests around). You don't need to
install a global opener, unless your code expects it -- it's just a
convenience (or an inconvenience, sometimes).

Actually, did the OP say proxy basic auth. was involved? Don't
recall. I've never needed it for proxies, but there seems to be a
ProxyBasicAuthHandler in urllib2, so I guess that's what you meant to
use, rather than HTTPBasicAuthHandler (which is for website auth., not
proxy auth).

So, after all that, you end up with:

opener = urllib2.build_opener(urllib2.ProxyBasicAuthHandler )
data = opener.open('ftp://ftp.gnu.org').read()

(I like to close the response explicitly, though)
John

Jul 18 '05 #4
py*******@Hotpop.com (Anand Pillai) writes:
jj*@pobox.com (John J. Lee) wrote in message news:<87************@pobox.com>...
py*******@Hotpop.com (Anand Pillai) writes:

The code obviously caters to the bottom-line, in case
the programmer does not want to worry about environment
variables, would like to install the handlers himself
and his proxy needs authentication.


Sure, I understand that. I was just elaborating, no criticism was
intended!

It seems people are frightened to use urllib2 (perhaps partly because
the documentation has lots of sections -- though they're all very
short), so I like to point out that most of the time it's just a
matter of:

import urllib2
urllib2.urlopen("http://example.com/")
And most of the rest of the time, the only complications above that
are passing a Request instead of a URL (for adding headers and passing
around requests as first-class objects), and building your own opener
with build_opener (for choosing what features you want and don't want,
like authentication -- and adding your own features, of course).
John
Jul 18 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: bmiras | last post by:
I've got a problem using urllib2 to get a web page. I'm going through a proxy using user/password authentification and i'm trying to get a page asking for a HTTP authentification. And I'm using...
4
by: Fuzzyman | last post by:
urllib2 (under windows) will auto-detect your proxy settings and use those. Normally that's a good thing (I guess), except when it's not ! How do I switch off this behaviour ? I'm behind a...
1
by: Ray Slakinski | last post by:
Hello, I have defined a function to set an opener for urllib2, this opener defines any proxy and http authentication that is required. If the proxy has authencation itself and requests an...
6
by: Alejandro Dubrovsky | last post by:
I see from googling around that this is a popular topic, but I haven't seen anyone saying "ah, yes, that works", so here it goes. How does one connect through a proxy which requires basic...
3
by: itay_k | last post by:
Hi, I am running the following simple code (just open connection to some https page with proxy): proxy= '666.179.227.666:80' proxy=urllib2.ProxyHandler({"https":'https://'+proxy}) opener =...
1
by: Alessandro Fachin | last post by:
I write this simply code that should give me the access to private page with htaccess using a proxy, i don't known because it's wrong... import urllib,urllib2 #input url...
1
by: cp.finances.gouv | last post by:
Hello all, I'm facing a strange behavior of urllib2 trying to access gmail account behind a proxy (Squid). The following works perfectly : wget --save-cookies cookies...
0
by: Astan Chee | last post by:
Hi, Im trying to implement the logic from http://www.hypothetic.org/docs/msn/general/http_connections.php to a simple python code using urllib2 and some parts of urllib. Im behind a http proxy...
6
by: Jack | last post by:
I'm trying to use a proxy server with urllib2. So I have managed to get it to work by setting the environment variable: export HTTP_PROXY=127.0.0.1:8081 But I wanted to set it from the code....
2
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and efficiency. While initially associated with cryptocurrencies...
0
by: antdb | last post by:
Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was proposed, which integrated multiple engines and...
0
hi
by: WisdomUfot | last post by:
It's an interesting question you've got about how Gmail hides the HTTP referrer when a link in an email is clicked. While I don't have the specific technical details, Gmail likely implements measures...
1
by: Matthew3360 | last post by:
Hi, I have been trying to connect to a local host using php curl. But I am finding it hard to do this. I am doing the curl get request from my web server and have made sure to enable curl. I get a...
0
by: Rahul1995seven | last post by:
Introduction: In the realm of programming languages, Python has emerged as a powerhouse. With its simplicity, versatility, and robustness, Python has gained popularity among beginners and experts...
2
by: Ricardo de Mila | last post by:
Dear people, good afternoon... I have a form in msAccess with lots of controls and a specific routine must be triggered if the mouse_down event happens in any control. Than I need to discover what...
1
by: Johno34 | last post by:
I have this click event on my form. It speaks to a Datasheet Subform Private Sub Command260_Click() Dim r As DAO.Recordset Set r = Form_frmABCD.Form.RecordsetClone r.MoveFirst Do If...
1
by: ezappsrUS | last post by:
Hi, I wonder if someone knows where I am going wrong below. I have a continuous form and two labels where only one would be visible depending on the checkbox being checked or not. Below is the...
0
by: jack2019x | last post by:
hello, Is there code or static lib for hook swapchain present? I wanna hook dxgi swapchain present for dx11 and dx9.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.