Hi all,
is there a way to modify the time a call of
urllib.open(... )
waits for an answer from the other side? Have a tool
which automatically checks a list of websites for
certain content. The tool "hangs" when one of the
contacted websites behaves badly and "never" answers...
Thanks and regards
-stefan- 11 2554
Stefan Palme wrote:
is there a way to modify the time a call of
urllib.open(... )
waits for an answer from the other side? Have a tool
which automatically checks a list of websites for
certain content. The tool "hangs" when one of the
contacted websites behaves badly and "never" answers...
I believe this can only be set globally:
import socket
socket.setdefau lttimeout(secon ds)
Peter
Uuuh this is no solution for me, because the
website-checking tool is part of a very very big
application running in an application server, so
globally setting the timeout may break a lot of
other things...
But when there is a "default timeout" (as indicated by
the method name) - isn't there a "per-socket timeout"
too?
-stefan-
I believe this can only be set globally:
import socket
socket.setdefau lttimeout(secon ds)
Peter
Stefan Palme wrote:
(top-posting undone)
[Peter]
>I believe this can only be set globally:
import socket socket.setdefa ulttimeout(seco nds)
[Stefan]
Uuuh this is no solution for me, because the
website-checking tool is part of a very very big
application running in an application server, so
globally setting the timeout may break a lot of
other things...
But when there is a "default timeout" (as indicated by
the method name) - isn't there a "per-socket timeout"
too?
Yes, but it isn't as easily available...
Perhaps you find some ideas here: http://mail.python.org/pipermail/pyt...ry/070897.html
Peter
>>[Peter]
>>I believe this can only be set globally:
import socket socket.setdef aulttimeout(sec onds)
[Stefan] ... But when there is a "default timeout" (as indicated by the method name) - isn't there a "per-socket timeout" too?
[Peter]
Yes, but it isn't as easily available...
Perhaps you find some ideas here:
http://mail.python.org/pipermail/pyt...ry/070897.html
Thanks, will have a look at this.
Regards
-stefan-
Stefan Palme wrote:
>>>[Peter] I believe this can only be set globally:
import socket socket.setde faulttimeout(se conds)
[Stefan] ... But when there is a "default timeout" (as indicated by the method name) - isn't there a "per-socket timeout" too?
[Peter] Yes, but it isn't as easily available...
Perhaps you find some ideas here:
http://mail.python.org/pipermail/pyt...ry/070897.html
Thanks, will have a look at this.
This has recently been discussed on python-dev. It's likely that certain
libraries will acquire a timeout keyword argument in the next release,
but only if someone is concerned enough to develop the appropriate
patches - the principle appears to be accepted.
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Blog of Note: http://holdenweb.blogspot.com
See you at PyCon? http://us.pycon.org/TX2007
Stefan Palme <kl*****@hora-obscura.dewrite s:
is there a way to modify the time a call of
urllib.open(... )
waits for an answer from the other side? Have a tool which
automatically checks a list of websites for certain content. The
tool "hangs" when one of the contacted websites behaves badly and
"never" answers...
Other than by using socket timeouts, at least in Un*x, you can also
use signal.alarm. You can only have one OS-provided alarm pending at
a time, so if you want multiple overlapping timeouts you have to
schedule them yourself with a single alarm.
Steve Holden <st***@holdenwe b.comwrites:
[...]
This has recently been discussed on python-dev. It's likely that
certain libraries will acquire a timeout keyword argument in the next
release, but only if someone is concerned enough to develop the
appropriate patches - the principle appears to be accepted.
I didn't read the recent python-dev thread, but ISTM this has been
true since the socket timeout feature was added (Python 2.3?).
Apparently so far nobody wanted it enough to actually implement it.
John
Paul Rubin <http://ph****@NOSPAM.i nvalidwrites:
Stefan Palme <kl*****@hora-obscura.dewrite s:
is there a way to modify the time a call of
urllib.open(... )
waits for an answer from the other side? Have a tool which
automatically checks a list of websites for certain content. The
tool "hangs" when one of the contacted websites behaves badly and
"never" answers...
Other than by using socket timeouts, at least in Un*x, you can also
use signal.alarm. You can only have one OS-provided alarm pending at
a time, so if you want multiple overlapping timeouts you have to
schedule them yourself with a single alarm.
Note that neither of these is guaranteed to time out urllib.open(),
though usually they will.
Another way to solve the underlying problem is not to try to time out
the network operation(s) at all, but to use threads or non-blocking
sockets (e.g. using Twisted).
John
>I believe this can only be set globally:
>> import socket socket.setdefa ulttimeout(seco nds)
StefanUuuh this is no solution for me, because the website-checking
Stefantool is part of a very very big application running in an
Stefanapplicati on server, so globally setting the timeout may break a
Stefanlot of other things...
Can you try this patch: http://python.org/sf/723312
? It's against current CVS. Also, it doesn't include urllib or urllib2
changes, just httplib. Getting this completed would be nice.
Skip This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: jeff |
last post by:
Hiya
im trying to pull tags off a website using python ive got a few things
running that have the potential to work its just i cant get them to
becuase of certain errors?
basically i dont what to download the images and all the stuff just
the html and then work from there, i think its timing out because its
trying to downlaod the images as well which i dont what to do as this
would decrease the speed of what im trying to achieve, the...
|
by: John Hunter |
last post by:
I have a test script below which I use to fetch urls into strings,
either over https or http. When over https, I use m2crypto.urllib and
when over http I use the standard urllib. Whenever, I import sockets
and setdefaulttimeout, however, using m2crypto.urllib tends to cause a
http.BadStatusLine to be raised, even if the timeout is set to be very
large. All of the documents in the test script can be accessed
publicly.
Any ideas? Is...
|
by: Pieter Edelman |
last post by:
Hi all,
I'm trying to submit some data using a POST request to a HTTP server with
BASIC authentication with python, but I can't get it to work. Since it's
driving me completely nuts, so here's my cry for help.
The server is an elog logbook server (http://midas.psi.ch/elog/). It is
protected with a password and an empty username. I can login both using
urllib and urllib2 (suppose the password is "foobar", the logbook is
running on port...
|
by: Timothy Wu |
last post by:
Hi,
I'm trying to fill the form on page
http://www.cbs.dtu.dk/services/TMHMM/ using urllib.
There are two peculiarities. First of all, I am filling in incorrect
key/value pairs in the parameters on purpose because that's the only
way I can get it to work.. For "version" I am suppose to leave it
unchecked, having value of empty string. And for name "outform" I am
suppose to assign it a value of "-short". Instead, I left out
|
by: Jaap Spies |
last post by:
Hi,
Running Fedora Core 4: Python 2.4.3 and Python 2.4.1.
I'm getting:
IOError: (2, 'No such file or directory')
all the time.
Trying to track down this problem:
Python 2.4.1 (#1, May 16 2005, 15:19:29)
| |
by: John Nagle |
last post by:
There's no way to set a timeout if you use "urllib" to open a URL.
"HTTP", which "urllib" uses, supports this, but the functionality
is lost at the "urllib" level.
It's not available via "class URLopener" or "FancyURLopener", either.
There is a non-thread-safe workaround from 2003 at
http://mail.python.org/pipermail/python-bugs-list/2003-September/020405.html
|
by: supercooper |
last post by:
I am downloading images using the script below. Sometimes it will go
for 10 mins, sometimes 2 hours before timing out with the following
error:
Traceback (most recent call last):
File "ftp_20070326_Downloads_cooperc_FetchLibreMapProjectDRGs.py",
line 108, i
n ?
urllib.urlretrieve(fullurl, localfile)
File "C:\Python24\lib\urllib.py", line 89, in urlretrieve
|
by: John Nagle |
last post by:
I thought I had all the timeout problems with urllib worked around,
but no.
socket.setdefaulttimeout is useful, but not always effective.
I'm setting that to 15 seconds.
If the host end won't open the connection within 15 seconds,
urllib times out. But if the host end opens the connection,
then never sends anything, urllib waits for many minutes before
timing out. Any idea how to deal with this? And don't just
say "use urllib2"...
|
by: John Nagle |
last post by:
urllib has a "hole" in its timeout protection.
Using "socket.setdefaulttimeout" will make urllib time out if a
site doesn't open a TCP connection in the indicated time. But if the site
opens the TCP connection and never sends HTTP headers, it takes about
20 minutes for the read in urllib's "open" to time out.
There are some web servers that produce this behavior, and
many seem to be associated with British universities and nonprofits....
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
| |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |