timeout in urllib.open()

[Stefan Palme]

Hi all,

is there a way to modify the time a call of

urllib.open(...)

waits for an answer from the other side? Have a tool
which automatically checks a list of websites for
certain content. The tool "hangs" when one of the
contacted websites behaves badly and "never" answers...

Thanks and regards
-stefan-

[Peter Otten]

Stefan Palme wrote:

is there a way to modify the time a call of

urllib.open(...)

waits for an answer from the other side? Have a tool
which automatically checks a list of websites for
certain content. The tool "hangs" when one of the
contacted websites behaves badly and "never" answers...

I believe this can only be set globally:

import socket
socket.setdefaulttimeout(seconds)

Peter

[Stefan Palme]

Uuuh this is no solution for me, because the
website-checking tool is part of a very very big
application running in an application server, so
globally setting the timeout may break a lot of
other things...

But when there is a "default timeout" (as indicated by
the method name) - isn't there a "per-socket timeout"
too?

-stefan-

I believe this can only be set globally:

import socket
socket.setdefaulttimeout(seconds)

Peter

[Peter Otten]

Stefan Palme wrote:

(top-posting undone)

[Peter]

>I believe this can only be set globally:

import socket
socket.setdefaulttimeout(seconds)

[Stefan]

Uuuh this is no solution for me, because the
website-checking tool is part of a very very big
application running in an application server, so
globally setting the timeout may break a lot of
other things...

But when there is a "default timeout" (as indicated by
the method name) - isn't there a "per-socket timeout"
too?

Yes, but it isn't as easily available...

Perhaps you find some ideas here:

http://mail.python.org/pipermail/pyt...ry/070897.html

Peter

[Stefan Palme]

>>[Peter]

>>I believe this can only be set globally:

import socket
socket.setdefaulttimeout(seconds)

[Stefan]
...
But when there is a "default timeout" (as indicated by
the method name) - isn't there a "per-socket timeout"
too?

[Peter]
Yes, but it isn't as easily available...

Perhaps you find some ideas here:

http://mail.python.org/pipermail/pyt...ry/070897.html

Thanks, will have a look at this.

Regards
-stefan-

[Steve Holden]

Stefan Palme wrote:

>>>[Peter]
I believe this can only be set globally:

import socket
socket.setdefaulttimeout(seconds)

[Stefan]
...
But when there is a "default timeout" (as indicated by
the method name) - isn't there a "per-socket timeout"
too?

[Peter]
Yes, but it isn't as easily available...

Perhaps you find some ideas here:

http://mail.python.org/pipermail/pyt...ry/070897.html

Thanks, will have a look at this.

This has recently been discussed on python-dev. It's likely that certain
libraries will acquire a timeout keyword argument in the next release,
but only if someone is concerned enough to develop the appropriate
patches - the principle appears to be accepted.

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Blog of Note: http://holdenweb.blogspot.com
See you at PyCon? http://us.pycon.org/TX2007

[Paul Rubin]

Stefan Palme <kl*****@hora-obscura.dewrites:

is there a way to modify the time a call of

urllib.open(...)

waits for an answer from the other side? Have a tool which
automatically checks a list of websites for certain content. The
tool "hangs" when one of the contacted websites behaves badly and
"never" answers...

Other than by using socket timeouts, at least in Un*x, you can also
use signal.alarm. You can only have one OS-provided alarm pending at
a time, so if you want multiple overlapping timeouts you have to
schedule them yourself with a single alarm.

[John J. Lee]

Steve Holden <st***@holdenweb.comwrites:
[...]

This has recently been discussed on python-dev. It's likely that
certain libraries will acquire a timeout keyword argument in the next
release, but only if someone is concerned enough to develop the
appropriate patches - the principle appears to be accepted.

I didn't read the recent python-dev thread, but ISTM this has been
true since the socket timeout feature was added (Python 2.3?).
Apparently so far nobody wanted it enough to actually implement it.
John

[John J. Lee]

Paul Rubin <http://ph****@NOSPAM.invalidwrites:

Stefan Palme <kl*****@hora-obscura.dewrites:

is there a way to modify the time a call of

urllib.open(...)

waits for an answer from the other side? Have a tool which
automatically checks a list of websites for certain content. The
tool "hangs" when one of the contacted websites behaves badly and
"never" answers...

Other than by using socket timeouts, at least in Un*x, you can also
use signal.alarm. You can only have one OS-provided alarm pending at
a time, so if you want multiple overlapping timeouts you have to
schedule them yourself with a single alarm.

Note that neither of these is guaranteed to time out urllib.open(),
though usually they will.

Another way to solve the underlying problem is not to try to time out
the network operation(s) at all, but to use threads or non-blocking
sockets (e.g. using Twisted).
John

[skip]

>I believe this can only be set globally:

>>
import socket
socket.setdefaulttimeout(seconds)

StefanUuuh this is no solution for me, because the website-checking
Stefantool is part of a very very big application running in an
Stefanapplication server, so globally setting the timeout may break a
Stefanlot of other things...

Can you try this patch:

http://python.org/sf/723312

? It's against current CVS. Also, it doesn't include urllib or urllib2
changes, just httplib. Getting this completed would be nice.

Skip

[John J. Lee]

sk**@pobox.com writes:

>I believe this can only be set globally:
>>
>import socket
>socket.setdefaulttimeout(seconds)

StefanUuuh this is no solution for me, because the website-checking
Stefantool is part of a very very big application running in an
Stefanapplication server, so globally setting the timeout may break a
Stefanlot of other things...

Can you try this patch:

http://python.org/sf/723312

? It's against current CVS. Also, it doesn't include urllib or urllib2
changes, just httplib. Getting this completed would be nice.

Great. Added some comments, will try to write tests and urllib2 patch
later.
John

[Facundo Batista]

Stefan Palme wrote:

is there a way to modify the time a call of

urllib.open(...)

waits for an answer from the other side? Have a tool

I'm working on adding a socket_timeout parametero to urllib2.urlopen.

Regards,

--
.. Facundo
..
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/