473,327 Members | 2,016 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,327 software developers and data experts.

urllib2 request blocks

I just tried this:
import urllib2
urllib2.urlopen( "https://passenger.ssc.com/~dmarti/contrib-faq/" )


It sits forever. Loading that URL in a normal browser gets me a popup asking
about server certificate authenticity, which presumably is what urllib2 is
blocking at.

How can I prepare my code for this eventuality and ensure it doesn't block?
Jul 18 '05 #1
2 2573
Derek Fountain wrote:
I just tried this:

import urllib2
urllib2.urlopen( "https://passenger.ssc.com/~dmarti/contrib-faq/" )

It sits forever. Loading that URL in a normal browser gets me a popup asking
about server certificate authenticity, which presumably is what urllib2 is
blocking at.

How can I prepare my code for this eventuality and ensure it doesn't block?


You can try to write the URL like that :

"https://username:pa******@passenger.ssc.com/~dmarti/contrib-faq/"
Jul 18 '05 #2
Derek Fountain <de*****@example.com> writes:
I just tried this:
import urllib2
urllib2.urlopen( "https://passenger.ssc.com/~dmarti/contrib-faq/" )

It sits forever. Loading that URL in a normal browser gets me a popup asking
about server certificate authenticity, which presumably is what urllib2 is
blocking at.


That script doesn't hang for me, using Python 2.3. I get a 200
response, with content the same as I see in Mozilla (Linux journal
author's guide and FAQ). The socket module does not verify the
authenticity of servers, so neither does urllib or urllib2.

How can I prepare my code for this eventuality and ensure it doesn't block?


Depends on why it's blocking.

sockets in 2.3 now support a timeout (that parameter is not fed
through to the httplib, urllib or urllib2 public interfaces, but you
can still set the default socket timeout in your application code).
If it's DNS that blocking, though, that won't help you. I think
proxies like squid do DNS lookups in a separate process (I guess you
can use threads too).

Alternatively, you can start messing with things like asyncore or
Twisted, or use threads (I don't recommend the latter :-/).
John
Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: bmiras | last post by:
I've got a problem using urllib2 to get a web page. I'm going through a proxy using user/password authentification and i'm trying to get a page asking for a HTTP authentification. And I'm using...
1
by: Doug Gray | last post by:
Python2.3 on Redhat Linux 8.0 here is the code. import urllib2 import ClientCookie request = urllib2.Request("http://fantasygames.sportingnews.com/crs/home_check_reg.htm
2
by: John F Dutcher | last post by:
Can anyone comment on why the code shown in the Python error is in some way incorrect...or is there a problem with Python on my hoster's site ?? The highlites don't seem to show here...but line...
1
by: Doug Farrell | last post by:
Hi all, I'm trying to build a web page crawler to help us build our websites, which are driven by static pages after they are called the first time. Anyway, I can use urllib2.urlopen() no...
0
by: jacob c. | last post by:
When I request a URL using urllib2, it appears that urllib2 always makes the request using HTTP 1.0, and not HTTP 1.1. I'm trying to use the "If-None-Match"/"ETag" HTTP headers to conserve...
4
by: Monty | last post by:
Hello, Sorry for this maybe stupid newbie question but I didn't find any answer in all my readings about python: With urllib, using urlretrieve, it's possible to get the number of blocks...
1
by: kelio | last post by:
I have a simple cgi-script on a server that prints all key-value pairs from a request. And it really works when i use a browser and type smth like http://server/cgi-bin/test?name=mike&johny=dummy....
7
by: itay_k | last post by:
Hi, I dont understand why this is so complicated, just to add one line of cookie header on the GET request. This is my unworking code: import time import Cookie import cookielib, urllib2
2
by: Joshua Kugler | last post by:
Standard disclaimer: read, googled, read some more. If you have a link, please free free to point me there. I'm using HTTPlib to construct some functional tests for a web app we're writing. ...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.