472,347 Members | 2,361 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,347 software developers and data experts.

urllib (in thread) never returns

Hash: SHA1


I've been having an intermittent problem with urllib.
With an interval of around 15 minutes (i.e.: this is run every 15m)
this code runs fine for about 1-2 weeks, but then gets it's
knickers in a twist, and never seems to return, nor except.

log("ABOUT TO USE urllib TO FETCH ["+self.url+"]")
f = urllib.urlopen(self.url)
temp_xml = f.read()
[1] log("DONE! GOT %d bytes for [%s]" % (len(str(temp_xml)),self.url))
refreshed = True
[2] log("Error: fetching ["+self.url+"], retrying ... ")

In the 'broken' state, I never see the log message [1] or [2], it
just sits in either the urlopen() or read() forever.

The code is running in a separate thread.
TCP timeout is set to 45 seconds.
Running on Linux (gentoo)

What I think is wrong, is that the server is sitting
behind a somewhat-dodgey ADSL connection. The server it's
contacting is also on a dodgey ADSL connection.

I'm guessing that the socket is opened with
the ADSL modem is NATing the connection, but when
the backend internet connection bounces, the NAT'ed
connection somehow stays up, leaving the connection
somehow dangling; alive, but dead too...

I would have thought that some urllib-internal timeout would
fix this?!

I could watch the thread from another thread, implementing
my own timeout... But then (AFAIK) there's no way to terminate
a mis-behaving thread anyways.

Any suggestions?

- -Kingsley
Version: GnuPG v1.4.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEut+ISo2jKyi4JlgRApIdAJ4iIWDLXR9am659XAS1aj Lv1ry12wCfeOiI
Jul 17 '06 #1
1 1451
>it just sits in either the urlopen() or read() forever.
>I would have thought that some urllib-internal timeout
would fix this?!
Yes, but you'll need to enable it. See socket.setdefaulttimeout() :

René Pijlman
Jul 17 '06 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: David Sworder | last post by:
This message was already cross-posted to C# and ADO.NET, but I forgot to post to this "general" group... sorry about that. It just occured to me...
by: Keyee Hsu | last post by:
Hi, I have a C# app that creates an AppDomain, enters it, and spawns an asyn thread to do some work and then block itself. Upon the completion of...
by: David Sworder | last post by:
Hi, I'm developing an application that will support several thousand simultaneous connections on the server-side. I'm trying to maximize...
by: Fernando Rodríguez | last post by:
Hi, Assume I have Main function that looks like this: static void Main(string args) { Action h1 = new Action(0);
by: Artem | last post by:
When I use the method Thread.Abort, it only sends a request of aborting to OS to stop a thread. The thread itself isn't killed and allocated...
by: Joe HM | last post by:
Hello - I have a function that calls Thread.Abort() to stop a thread in a _Closed() Method of a GUI. The thread contains a blocking call on a...
by: =?Utf-8?B?R3JlZyBMYXJzZW4=?= | last post by:
I'm trying to figure out how to modify a panel (panel1) from a backgroundworker thread. But can't get the panel to show the new controls added by...
by: Tomaz Koritnik | last post by:
Is it possible to asynchronously call a method from worker thread so that this method would execute in main-thread? I'm doing some work in worker...
by: Boki | last post by:
Hi All, When thread returns, that means the thread is closed ? What is the standard way to kill it ? threadA= null ; Thank you for advice!...
by: liu yang | last post by:
I want to build a timer for each thread, and the timer must be thread- safe. How to do that? My environment is Linux and Pthread. Thanks.
by: concettolabs | last post by:
In today's business world, businesses are increasingly turning to PowerApps to develop custom business applications. PowerApps is a powerful tool...
by: better678 | last post by:
Question: Discuss your understanding of the Java platform. Is the statement "Java is interpreted" correct? Answer: Java is an object-oriented...
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and...
by: jalbright99669 | last post by:
Am having a bit of a time with URL Rewrite. I need to incorporate http to https redirect with a reverse proxy. I have the URL Rewrite rules made...
by: Matthew3360 | last post by:
Hi there. I have been struggling to find out how to use a variable as my location in my header redirect function. Here is my code. ...
by: Arjunsri | last post by:
I have a Redshift database that I need to use as an import data source. I have configured the DSN connection using the server, port, database, and...
by: WisdomUfot | last post by:
It's an interesting question you've got about how Gmail hides the HTTP referrer when a link in an email is clicked. While I don't have the specific...
by: Matthew3360 | last post by:
Hi, I have been trying to connect to a local host using php curl. But I am finding it hard to do this. I am doing the curl get request from my web...
by: Oralloy | last post by:
Hello Folks, I am trying to hook up a CPU which I designed using SystemC to I/O pins on an FPGA. My problem (spelled failure) is with the...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.