469,890 Members | 1,411 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,890 developers. It's quick & easy.

urllib (in thread) never returns

Hash: SHA1


I've been having an intermittent problem with urllib.
With an interval of around 15 minutes (i.e.: this is run every 15m)
this code runs fine for about 1-2 weeks, but then gets it's
knickers in a twist, and never seems to return, nor except.

log("ABOUT TO USE urllib TO FETCH ["+self.url+"]")
f = urllib.urlopen(self.url)
temp_xml = f.read()
[1] log("DONE! GOT %d bytes for [%s]" % (len(str(temp_xml)),self.url))
refreshed = True
[2] log("Error: fetching ["+self.url+"], retrying ... ")

In the 'broken' state, I never see the log message [1] or [2], it
just sits in either the urlopen() or read() forever.

The code is running in a separate thread.
TCP timeout is set to 45 seconds.
Running on Linux (gentoo)

What I think is wrong, is that the server is sitting
behind a somewhat-dodgey ADSL connection. The server it's
contacting is also on a dodgey ADSL connection.

I'm guessing that the socket is opened with
the ADSL modem is NATing the connection, but when
the backend internet connection bounces, the NAT'ed
connection somehow stays up, leaving the connection
somehow dangling; alive, but dead too...

I would have thought that some urllib-internal timeout would
fix this?!

I could watch the thread from another thread, implementing
my own timeout... But then (AFAIK) there's no way to terminate
a mis-behaving thread anyways.

Any suggestions?

- -Kingsley
Version: GnuPG v1.4.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEut+ISo2jKyi4JlgRApIdAJ4iIWDLXR9am659XAS1aj Lv1ry12wCfeOiI
Jul 17 '06 #1
1 1340
>it just sits in either the urlopen() or read() forever.
>I would have thought that some urllib-internal timeout
would fix this?!
Yes, but you'll need to enable it. See socket.setdefaulttimeout() :

René Pijlman
Jul 17 '06 #2

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

10 posts views Thread by Fernando Rodríguez | last post: by
8 posts views Thread by =?Utf-8?B?R3JlZyBMYXJzZW4=?= | last post: by
12 posts views Thread by Tomaz Koritnik | last post: by
1 post views Thread by Waqarahmed | last post: by
reply views Thread by Salome Sato | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.