473,320 Members | 1,948 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

urllib(2) and https blues? try pytunnel for python tunnelling

The python libraries like urllib and httplib do not support ssl
through a proxy.
Urllib2 supports http through a proxy or https alone, but not https
through a proxy.

A while ago my wife complained to me that perl had trouble w/https. I
was surprised since perl has a lot of networking libraries. Preferring
python, I took a look a python and found the same problem. Eventually
I figured out what python was missing: tunnelling. Though she may
_coax_ me to port the code to perl, I'm writing it first in my
favorite language.

Pytunnel offers a tunnel that can be used for (among other things)
tunnelling the ssl through a proxy.

I'd be willing to offer suggestions or myself add the necessary code
to urllib2. Anyone know how one goes about doing either?

Your python code does not need to be changed except that the port and
ip are supplied by the tunnel. Pytunnel uses a custom recvall which is
like sendall but for recv. Recvall uses non-blocking sockets,timeouts
and sleeps to attempt to get all of the data. If you have a bad
network connection you'd want to set the timeout to be high. Recvall
will probably be slower than a standard recvall.

The code is rough and needs better error handling, comments and such.
But since I got it working I decided to post it.

You can find it at:

http://aspn.activestate.com/ASPN/Coo.../Recipe/213238
Example of usage:

import pytunnel,httplib

def tunnel_this(ip,port):
conn = httplib.HTTPSConnection(ip,port=port)
conn.putrequest('GET', '/')
conn.endheaders()
response = conn.getresponse()
print response.read()

tunnel=pytunnel.build(host='login.yahoo.com',proxy _host='h1',proxy_user='u',proxy_pass='p')
tunnel.run(tunnel_this)

Jul 18 '05 #1
1 3053
According to john <py*****@yahoo.com>:
The python libraries like urllib and httplib do not support ssl
through a proxy.
Urllib2 supports http through a proxy or https alone, but not https
through a proxy.


M2Crypto's contrib/ contains an isaac.httpslib.py, with the following
blurb:

This is Isaac Salzberg's application of Mihai Ibanescu's patch
(available on SF) that allows HTTPS tunneling through an
authenticating proxy.

This one's a double whammy: it works with IIS through the
authenticating proxy, whereas the one on SF, which uses Python's
built-in SSL, doesn't.

This code is not folded into the main distribution because:

1. Apparently Mihai is still working on it.
2. Mihai uses Python's built-in SSL. Isaac patched it to use
M2Crypto.SSL. The stuff is essentially #ifdef'ed code.
3. I don't have an authenticating proxy nor an IIS server to test
against, so I can't clean up the code. Volunteers welcome. ;-)

Thanks Isaac.
--
Ng Pheng Siong <ng**@netmemetic.com>

http://firewall.rulemaker.net -+- Manage Your Firewall Rulebase Changes
http://www.post1.com/home/ngps -+- Open Source Python Crypto & SSL
Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Gary Feldman | last post by:
I think I've found a deficiency in the design of urllib related to https. In order to complete an https connection, it appears that URLOpener and hence FancyURLOpener require the key and cert...
3
by: Volker M. | last post by:
Hey, I want to open a list of URLs with Pythons urllib and the fuction open(URL) automatically. It is important that the program open ONLY normal http-sites and no https-sites with...
1
by: John Hunter | last post by:
I have a test script below which I use to fetch urls into strings, either over https or http. When over https, I use m2crypto.urllib and when over http I use the standard urllib. Whenever, I...
3
by: bowman.joseph | last post by:
Hi, I'm new to python. I've been handed the job of modifying a script we have here at work that pulls content from a zope site, to create static html. They wanted a check to make sure the...
0
by: Ali.Sabil | last post by:
hello all, I just maybe hit a bug in both urllib and urllib2, actually urllib doesn't support proxy authentication, and if you setup the http_proxy env var to...
0
by: Cecil Westerhof | last post by:
I have a strange problem. I wrote a script that uses urllib.urlopen to fetch a page through https. In Python 2.2.2 this works without a problem. But when I use the script in Python I get: ...
4
by: kgrafals | last post by:
Hi, I'm just trying to read from a webpage with urllib but I'm getting IOErrors. This is my code: import urllib sock = urllib.urlopen("http://www.google.com/") and this is the error:
4
by: John Nagle | last post by:
There's no way to set a timeout if you use "urllib" to open a URL. "HTTP", which "urllib" uses, supports this, but the functionality is lost at the "urllib" level. It's not available via "class...
0
by: Ghirai | last post by:
On Saturday 16 August 2008 12:16:14 Fredrik Lundh wrote: Thanks, that seems to be getting me very close of what i need. -- Regards, Ghirai.
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.