473,701 Members | 2,603 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

No way to set a timeout in "urllib".

There's no way to set a timeout if you use "urllib" to open a URL.
"HTTP", which "urllib" uses, supports this, but the functionality
is lost at the "urllib" level.

It's not available via "class URLopener" or "FancyURLopener ", either.

There is a non-thread-safe workaround from 2003 at

http://mail.python.org/pipermail/pyt...er/020405.html

but it was rejected as a feature at

https://sourceforge.net/tracker/?fun...&group_id=5470

without anything better going in. Despite this, current documentation
recommends that approach:

http://svn.python.org/projects/pytho...to/urllib2.rst

Someone proposed to fix this

http://mail.python.org/pipermail/pyt...ly/066967.html

but was discouraged from doing so.

The code was forked by Zope as a workaround in 2003:

http://pywebsvcs.sourceforge.net/api...s/Utility.html

but that's not in the mainstream Python tree.

The correct fix would probably be to add methods to class
URLopener to control this; that's the usual way of handling special
URL opening situations.

John Nagle
Dec 29 '06 #1
4 6135

JohnThere's no way to set a timeout if you use "urllib" to open a URL.
John"HTTP", which "urllib" uses, supports this, but the functionality
Johnis lost at the "urllib" level.

John It's not available via "class URLopener" or "FancyURLopener ",
John either.

JohnThere is a non-thread-safe workaround from 2003 at

...

This topic has come up several times since timeouts were added to socket.
Each time we've asked for a patch that adds timeouts in a rational manner to
all the stuff layered on top of the socket module (httplib, ftplib, etc). As
far as I know it's apparently never been important enough for anyone to rise
to the challenge. If I remember next spring perhaps I'll submit it as a
possible Google Summer of Code proposal.

JohnThe correct fix would probably be to add methods to class
JohnURLopener to control this; that's the usual way of handling
Johnspecial URL opening situations.

The correct way would be to deal with it at the httplib level, then
percolate it up to urllib2. Urllib should probably not be extended any
further.

Skip

Dec 30 '06 #2
sk**@pobox.com wrote:
JohnThere's no way to set a timeout if you use "urllib" to open a URL.
John"HTTP", which "urllib" uses, supports this, but the functionality
Johnis lost at the "urllib" level.

John It's not available via "class URLopener" or "FancyURLopener ",
John either.

JohnThere is a non-thread-safe workaround from 2003 at

...

This topic has come up several times since timeouts were added to socket.
Each time we've asked for a patch that adds timeouts in a rational manner to
all the stuff layered on top of the socket module (httplib, ftplib, etc). As
far as I know it's apparently never been important enough for anyone to rise
to the challenge. If I remember next spring perhaps I'll submit it as a
possible Google Summer of Code proposal.
It ought not to take more than a day or two, although it is annoying
that you have to deal with the problem at several levels.

If you're looking for a Summer of Code project, consider finishing
the Python SSL library, which doesn't even check certificates, let alone
revocation lists. It's a pure interface problem. OpenSSL does all the
hard parts, but the Python glue code is crude and incomplete.

John Nagle
Dec 30 '06 #3

JohnIf you're looking for a Summer of Code project, ...

I'm not. I'm about 25 years out of grad school. ;-)

Skip
Dec 30 '06 #4
John Nagle <na***@animats. comwrites:
There's no way to set a timeout if you use "urllib" to open a URL.
"HTTP", which "urllib" uses, supports this, but the functionality
is lost at the "urllib" level.

It's not available via "class URLopener" or "FancyURLopener ", either.

There is a non-thread-safe workaround from 2003 at

http://mail.python.org/pipermail/pyt...er/020405.html

but it was rejected as a feature at

https://sourceforge.net/tracker/?fun...&group_id=5470

without anything better going in. Despite this, current documentation
recommends that approach:

http://svn.python.org/projects/pytho...to/urllib2.rst
And...? What specifically are you complaining about?

Just as a matter of fact (I'm not grumpy about it): I see from your
message that you already know that there is not a shortage of people
who spot the lack of this kind of feature. The shortage is of people
who will actually do the job of adding the feature -- most
importantly, people who will contribute high quality patches,
including tests and docs, and people who will review other people's
patches.

Someone proposed to fix this

http://mail.python.org/pipermail/pyt...ly/066967.html

but was discouraged from doing so.
[...]

While it might be discouraging to be confronted with tiresome
realities of schedules and resources, those are the realities.

The flip side is that you can make a real difference by putting in
some work.

Looking forward to your patch :-)
John
Dec 30 '06 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
3048
by: Gilles Lenfant | last post by:
Hi, I make an app where I need to convert HTML to text in a "clever" way (means it tries to mimic when possible a browser rendering). Actually I spawn with popen2 a "lynx" that makes a perfect job. But I need a 100% pythonic stuff to have my app (Zope product) running on non Unix boxes. Any hint ?
2
6279
by: Eino Mäkitalo | last post by:
It seems that urrlib2 default redirection does not allow me to handle Cookies. Service I'm trying seems to use IP switcher and session id's with cookies. After successful login it changes session id (PD-H_SESSION-ID) in 302 Moved temporarily. Urllib2 is so clever that it handles redirection but with wrong cookies. Some hookings? Just own version from source. What is the most lazy way to handle this. Older urllib?
27
3074
by: Ron Adam | last post by:
There seems to be a fair amount of discussion concerning flow control enhancements lately. with, do and dowhile, case, etc... So here's my flow control suggestion. ;-) It occurred to me (a few weeks ago while trying to find the best way to form a if-elif-else block, that on a very general level, an 'also' statement might be useful. So I was wondering what others would think of it.
10
2265
by: Pete | last post by:
I have the following code: <built-in method close of file object at 0xb7cc76e0> The file "temp.html" is created, but it doesn't look like the page at www.python.org. I'm guessing there are multiple frames and my code did not get everything. Can anyone point me to a tutorial or other reference on how to "get" all of the html contents at a particular page?
1
2262
by: Joe Peterson | last post by:
I've been doing a lot of searching on the topic of one of Python's more disturbing issues (at least to me): the fact that if a __del__ finalizer is defined and a cyclic (circular) reference is made, the garbage collector cannot clean it up. First of all, it seems that it's best to avoid using __del__. So far, I have never used it in my Python programming. So I am safe there. Or am I? Also, to my knowledge, I have never created a...
1
1668
by: John Nagle | last post by:
I was looking at the code for "urllib", and there's some undocumented "FTP cacheing" code in there that's not thread safe. The documentation for "urllib" Is there any good reason to keep that code in "urllib"? Unless something critical uses that undocumented feature, that code probably ought to come out. Comments? John Nagle
8
1418
by: John Nagle | last post by:
The Python documentation for "str" says "str() : Return a string containing a nicely printable representation of an object." However, there's no mention of the fact that "str" of a Unicode string with non-ASCII characters will raise a conversion exception. The documentation (and several Python books) seem to indicate that "str" will produce some "printable representation" for anything, not raise an exception. I know, it was proposed...
4
3771
by: =?utf-8?B?Qm9yaXMgRHXFoWVr?= | last post by:
Hello, (sorry to begin with Java in a Python list ;-) in Java, when I want to pass input to a function, I pass "InputStream", which is a base class of any input stream. In Python, I found that "file" objects exist. While specifying argument types in Python is not possible as in Java, it is possible to check whether an object is an instance of some class and that's what I need - I need to check if an argument is a "file"-like object,...
2
2623
by: John Nagle | last post by:
For some reason, Python's parser for "robots.txt" files doesn't like Wikipedia's "robots.txt" file: False The Wikipedia robots.txt file passes robots.txt validation, and it doesn't disallow unknown browsers. But the Python parser doesn't see it that way. No matter what user agent or URL is specified; for that robots.txt file, the only answer is "False". It's failing in Python 2.4 on Windows and 2.5 on Fedora Core.
0
8736
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9083
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8977
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8934
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7824
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4662
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
3102
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2398
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
2035
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.