473,327 Members | 1,976 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,327 software developers and data experts.

socket timeout / m2crypto.urllib problems


I have a test script below which I use to fetch urls into strings,
either over https or http. When over https, I use m2crypto.urllib and
when over http I use the standard urllib. Whenever, I import sockets
and setdefaulttimeout, however, using m2crypto.urllib tends to cause a
http.BadStatusLine to be raised, even if the timeout is set to be very
large. All of the documents in the test script can be accessed
publicly.

Any ideas? Is there a better/easier way to get https docs in python?

Thanks,
JDH

import urllib, socket
from cStringIO import StringIO
from M2Crypto import Rand, SSL, m2urllib

#comment out this line and the script generally works, but without it
#my zope process, which is using this code, hangs.
socket.setdefaulttimeout(200)
def url_to_string(source):
"""
get url as string, for https and http
"""
if source.startswith('https:'):
sh = StringIO()
url = m2urllib.FancyURLopener()
url.addheader('Connection', 'close')
u = url.open(source)

while 1:
data = u.read()
if not data: break
sh.write(data)
return sh.getvalue()
else:
return urllib.urlopen(source).read()

if __name__=='__main__':
s1 = url_to_string('https://crcdocs.bsd.uchicago.edu/crcdocs/Files/informatics.doc')

s2 = url_to_string('http://yahoo.com')

s3 = url_to_string('https://crcdocs.bsd.uchicago.edu/crcdocs/Files/facepage.doc')
print len(s1), len(s2), len(s3)

Jul 18 '05 #1
1 2465
John Hunter <jd******@ace.bsd.uchicago.edu> writes:
[...]
Any ideas? Is there a better/easier way to get https docs in python?

[...]

Python 2.3 has https support built-in even on Windows.
John
Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Gary Feldman | last post by:
I think I've found a deficiency in the design of urllib related to https. In order to complete an https connection, it appears that URLOpener and hence FancyURLOpener require the key and cert...
7
by: Carl Waldbieser | last post by:
I tried to adapt the instructions for building the M2Crypto module (http://sandbox.rulemaker.net/ngps/m2/INSTALL.html) to build a version compatible with Python2.3, but I've had some mixed results....
2
by: Dilip | last post by:
Gurus I didn't personally encounter these problems but some friends of mine who are doing some deep heavy-metal C#/Networking application have run into these problems. Does anybody have any...
0
by: Jaap Spies | last post by:
Hi, Running Fedora Core 4: Python 2.4.3 and Python 2.4.1. I'm getting: IOError: (2, 'No such file or directory') all the time. Trying to track down this problem: Python 2.4.1 (#1, May 16...
4
by: kgrafals | last post by:
Hi, I'm just trying to read from a webpage with urllib but I'm getting IOErrors. This is my code: import urllib sock = urllib.urlopen("http://www.google.com/") and this is the error:
8
by: John Nagle | last post by:
I've been running M2Crypto successfully using Python 2.4 on Windows 2000, and now I'm trying to get it to work on Python 2.3.4 on Linux. Attempting to initialize a context results in Traceback...
10
by: John Nagle | last post by:
Here are three network-related exceptions. These were caught by "except" with no exception type, because none of the more specific exceptions matched. This is what a traceback produced: 1....
2
by: Heikki Toivonen | last post by:
M2Crypto has some old code that gets and sets socket timeouts in http://svn.osafoundation.org/m2crypto/trunk/M2Crypto/SSL/Connection.py, for example: def get_socket_read_timeout(self): return...
0
by: Heikki Toivonen | last post by:
The 0.19.1 release fixes the build when OpenSSL has been configured without EC support, thanks to Miloslav Trmac. M2Crypto is the most complete Python wrapper for OpenSSL featuring RSA, DSA, DH,...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.