470,833 Members | 1,424 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,833 developers. It's quick & easy.

socket timeout / m2crypto.urllib problems

I have a test script below which I use to fetch urls into strings,
either over https or http. When over https, I use m2crypto.urllib and
when over http I use the standard urllib. Whenever, I import sockets
and setdefaulttimeout, however, using m2crypto.urllib tends to cause a
http.BadStatusLine to be raised, even if the timeout is set to be very
large. All of the documents in the test script can be accessed

Any ideas? Is there a better/easier way to get https docs in python?


import urllib, socket
from cStringIO import StringIO
from M2Crypto import Rand, SSL, m2urllib

#comment out this line and the script generally works, but without it
#my zope process, which is using this code, hangs.
def url_to_string(source):
get url as string, for https and http
if source.startswith('https:'):
sh = StringIO()
url = m2urllib.FancyURLopener()
url.addheader('Connection', 'close')
u = url.open(source)

while 1:
data = u.read()
if not data: break
return sh.getvalue()
return urllib.urlopen(source).read()

if __name__=='__main__':
s1 = url_to_string('https://crcdocs.bsd.uchicago.edu/crcdocs/Files/informatics.doc')

s2 = url_to_string('http://yahoo.com')

s3 = url_to_string('https://crcdocs.bsd.uchicago.edu/crcdocs/Files/facepage.doc')
print len(s1), len(s2), len(s3)

Jul 18 '05 #1
1 2330
John Hunter <jd******@ace.bsd.uchicago.edu> writes:
Any ideas? Is there a better/easier way to get https docs in python?


Python 2.3 has https support built-in even on Windows.
Jul 18 '05 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

4 posts views Thread by Gary Feldman | last post: by
7 posts views Thread by Carl Waldbieser | last post: by
reply views Thread by Jaap Spies | last post: by
4 posts views Thread by kgrafals | last post: by
8 posts views Thread by John Nagle | last post: by
reply views Thread by Heikki Toivonen | last post: by
reply views Thread by mihailmihai484 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.